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Advance Praise for DevOps for Digital Leaders 


"Everyone on the DevOps journey wants to see how others solved similar 
problems in different business contexts. This book does a great job in chroni- 
cling how organizations have applied DevOps patterns and articulating the 
fantastic value they've created.” 


—Gene Kim, co-author of The Phoenix Project and DevOps Handbook 


“DevOps for Digital Leaders addresses an important gap in the literature, pre- 
senting DevOps from a business and organizational perspective rather than 
the more common technology and process angles. Aruna, Kieren, and Peter 
have done a great job of assembling, presenting, and addressing the core chal- 
lenges of a transition to DevOps. Not only do they look at the impact on the 
"path to production" of testing, deployment, and operational support, but they 
tackle the complex topic of API design and even present the ROI justification 
for DevOps, citing Dr. Nicole Forsgren's groundbreaking work with the State 
of DevOps reports. This book should be required reading for technology 
leaders wondering about the impact DevOps will have on their organization." 


—Dan North, DevOps and Continuous Delivery pioneer, 
Principal at Dan North & Associates Ltd. 


“If DevOps for Digital Leaders had existed several years ago when we started 
on our DevOps journey, it would have made our transition significantly easier. 
This book is an excellent resource for both experienced and new DevOps 
practitioners—more than just a 'how-to', it provides valuable insights and 
real-world examples to which everyone can relate.” 


—Dana Edwards, CTO/EVP MUFG Union Bank 


"As DevOps has risen to the top of the mind with IT departments big and small, 
new and old, there has been much written about the subject. Unfortunately, 
too much of it has been about *what is DevOps' or shrouded in almost a 
Zen-like philosophy. What has been missing is the practical ‘why’ and ‘how 
to' for practitioners. What's missing is a guide that will grow dog-eared and 
highlighted, with real-world examples and advice that practitioners will carry 
with them and go back to again and again. DevOps for Digital Leaders is that 
book. If you are looking for real-world examples and guidance on turning your 
organization into a high-performing IT operation, this is a must-have book for 
your library.” 


—Alan Shimel, Editor-in-Chief, DevOps.com 


“DevOps is no longer just a concept. This book by Aruna, Kieran, and Peter 
places DevOps squarely in the real world. It is a comprehensive guide to help 
you discover the origins of DevOps, realize the challenges, and gain a practi- 
cal understanding of where DevOps fits into your development program, IT 
operations, and business. It's a must-have book for the business side to truly 
comprehend DevOps, but also a great read for the IT community to under- 
stand how their work ultimately impacts the business. And most importantly, 
the book is loaded with valuable, actionable tips to help you navigate your 
DevOps journey.” 


—Pete Goldin, Editor and Publisher, APMdigest and DEVOPSdigest 


I would like to dedicate my first book to my mother, 
Sampurnam Thyagarajan, who has always been my 
inspiration, who stood by me every step of the way. 
I am who I am today because of her unconditional 
love and dedication. I love you Amma. 


—Aruna 


I dedicate this book to all the true IT believers and 
practitioners. Only from your collective effort can 
organizations build the generative DevOps culture 
needed to shape success. You know who you are! 


—Peter 


I dedicate this book to those who work to make the 
world a better place for future generations. 


— Kieran 
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Foreword 


As a software executive who's worn many hats across sales, services, develop- 
ment, support, operations, and even IT, | have a fairly unique perspective when 
| talk to business leaders about their technology strategies. Though the name 
DevOps didn't exist until fairly recently, the need for partnership between 
development and operations teams has always been around. Today DevOps 
has shifted from an emerging movement to a critical component of most 
enterprises’ digital transformation strategy. In other words, DevOps is moving 
from the Kanban board to the board of directors. 


While the idea behind this kind of collaboration is common sense, there are 
definitely challenges to driving mainstream adoption for DevOps. In my con- 
versations with C-suite executives, they invariably ask for two things: docu- 
mented best practices and actual case studies that they can take to their 
internal stakeholders. 


That's why this book is so relevant right now. It's more than just theory or a 
summary of the ideal DevOps toolchain; it's a set of practical insights and bet- 
ter practices that came about through many, many interviews and interactions 
with frontline practitioners about what works and what doesn't work in the 
real world. 


Aruna, Kieran, and Peter are career high tech marketers who shed light on 
the cultural and technological challenges to driving adoption for a DevOps 
philosophy across the software development lifecycle. With this book, they 
were able to bring together solid, actionable advice for anyone beginning their 
DevOps journey, including recommendations on how to measure success and 
return on investment from a business standpoint. 


The days of organizational silos are coming to an end. Communication, col- 
laboration, and automation are blurring the lines between development, QA, 
and IT Operations—and enterprises are embracing agile and DevOps prac- 
tices to increase software release frequency while improving overall quality. 
| hope that in reading this, you find ways to optimize your technology strategy, 
investments, and business outcomes. 


—Adam Elster, President of Global Field Operations, CA Technologies 
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DevOps: Conflict 
to Collaboration 


CHAPTER 


i 


DevOps in the 
Ascendency 


“Be ahead of the times through endless creativity, inquisitiveness, and 
pursuit of improvement.” 


—The Toyota Precepts' 


Accelerating Agile Practices in Today’s 
Software Factory 


In 2016, Formula | (FI) racecars, the ultimate in four-wheeled technology, are 
awash in wireless sensors and transmitters. 


Running your eyes across the graceful, sculpted bodywork of these 200+ mph 
engineering wonders, the only discernible interruption of their low-slung car- 
bon fiber noses are small clusters of wireless antennas jutting up directly into 
the drivers’ sightlines. 


“Toyota Quotes," The New York Times, Feb. 10, 2008: http: //www.nytimes.com/ 
2008/02/10/business/worldbusiness/10iht-10facts.9900222.html?_r=0 
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As subtle as these transmitters may be, why would designers, who spend end- 
less time and resources attempting to improve the aerodynamics and drivabil- 
ity of the machines, accept such a stark concession? The answer is simple—to 
arm their teams with priceless real-time data used to continuously improve 
performance. 


Along the pit row wall, in the trackside garages, and back in the very design 
centers where Fl engineers continue to refine their handiwork, telemetric 
information provided by those tiny antennae is immediately translated into 
change. 


While team principals and technical directors sitting directly adjacent to the 
track communicate directly with the drivers, advising them how to adjust set- 
tings on the fly, their colleagues in the garages prepare to adjust everything 
from tires to aerodynamics when the machines pull into the pits. 


Meanwhile, back at the factory, live data is consumed by all manner of engineer- 
ing teams who continually produce and modify new components to be utilized 
in subsequent competitions—beginning the process of further advancement 
even before the current race is complete. The push for innovation in Fl is a 
perpetual activity; in this sense, perhaps more so than in any other sport, the 
only constant is technological change. 


That Toyota Motor Corp., the global automotive giant best known for advanc- 
ing such data-driven, process-centric "just in time" manufacturing practices— 
concepts that revolutionized mass production of passenger cars—had such 
a short-lived and disappointing record in Fl can only be classified as ironic. 
However, the Kanban and Kata methodologies leveraged by Toyota since its 
inception, paralleled on a smaller, more specialized scale by today's Fl teams, 
stand as the most widely emulated management frameworks in history. 


With an emphasis on the adoption and evolution of processes designed to 
optimize resources while increasing production speed and quality—and most 
importantly promoting constant innovation—the "Toyota Way"? represents 
arguably the leading model for continuous improvement. 


In the words of W. Edwards Deming—the legendary American engineer and 
management consultant who helped spearhead the reinvention of Japanese 
industry after WW2—Toyota’s approach embodies the notion that, “It is not 
enough to do your best; you must know what to do, and then do your best.”? 


Grounded in constant observation and measurement of efficiency—from sup- 
ply chain management through final production, with the goal of constantly 
improving output, including and in particular worker productivity—this meth- 


?Toyota Motor Corporation Annual Report, 2003, page 19. 
“Made in Japan, MadeinWyoming.com, Rena Delbridge, copyright 1995: http:// 
madeinwyoming.net/profiles/deming.php 
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odology flies in the face of traditional “waterfall” workflows. American auto- 
maker Henry Ford may be credited with revolutionizing the assembly line, but 
Toyota is widely recognized for decoupling and perfecting it. 


Rather than creating linear dependencies, where each successive activity is 
wholly dependent on completion of the task preceding it, this revolutionary 
approach emphasizes techniques wherein production is dynamically adapted 
on the fly to result in optimal efficiency and maximum quality. 


Further illustrating the staunch devotion to continuous improvement repre- 
sented both in the Toyota Way and in his personal doctrine, Deming famously 
said, “It is not necessary to change. Survival is not mandatory.” 


This observation is neither subtle, nor, given industrial history, seemingly 
misplaced. 


Embracing DevOps in the Application Economy 


In today’s rapidly evolving Application Economy, it is widely recognized that, 
driven by the evolution of digital channels—across both the business-to-busi- 
ness and consumer segments—many organizations are reinventing or recast- 
ing themselves as providers of software and digital services. 


For example, the global population of mobile banking users is forecast to 
double to 1.8 billion by 2020, representing over 25 percent of the world’s 
population, according to KPMG.° As a result, banks are increasingly focused 
on advancement of web-based and mobile applications, versus expansion of 
brick-and-mortar operations. 


Furthermore, as business delivery mechanisms shift to the digital landscape, 
end users' tolerance of latency of such applications has grown increasingly 
narrow. According to a study published by Wired in June 2014, roughly 50 
percent of consumers expect a web page to load in two seconds or less—or 
they move to abandon it. 


Given this transformation, as traditional business services are replaced by 
largely web-based and mobile-friendly applications, organizations are being 
forced to completely reexamine their software development and IT manage- 
ment practices. Technology is no longer viewed as a supporting dependency, 
but rather as a primary element of conducting business. 


^Edward Deming, Out of the Crisis, (Cambridge, MA, MIT Press, 1982), p. 227. 

“Global Mobile Banking Report," KPMG LLP, copyright 2015: http: //www.kpmg.com/ 
channelislands/en/IssuesAndInsights/ArticlesPublications/Documents/ 
Digital%200fferings%20in%20mobile%20banking%20-%20May%202015. pdf 

*“Great Expectations: 47% of Consumers Want a Web Page to Load in Two Seconds 
or Less" by Nilesh Patel, Wired, copyright 2014: http://insights.wired.com/ 
profiles/blogs/47-of-consumers-expect-a-web-page-to-load-in-2-seconds- 
or-less#ixzz40vkvFwV1 
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Within that context, most of today's organizations are moving aggressively to 
adopt more agile, efficient software delivery and IT management practices to 
meet customers' evolving expectations. Among the fastest growing and most 
widely adopted strategies, aimed at bringing Toyota-like efficiency to the world 
of the applications lifecycle, is the methodology known as DevOps. 


Whether or not Patrick Debois understood that he was creating, or at the 
very least putting a name to, the movement that would completely rede- 
fine the manner in which organizations build and support software is unclear. 
What is known is that since Debois, a systems administrator, and a handful 
of like-minded software development and IT operations experts first coined 
the DevOps moniker in 2009, the concept has become a global phenomenon. 


The underlying concepts encompassed by DevOps are, at first glance, straight- 
forward, but represent seismic reformulation within the context of software 
production and support. Rather than maintaining discreet applications engi- 
neering ("Dev") and IT management (“Ops”) competencies and organiza- 
tions, DevOps dictates use of smaller teams with cross-functional expertise 
to improve software functionality and the processes used to deliver it. 


As highlighted in the seminal DevOps novel, The Phoenix Project, this mindset 
eliminates the fragmented approach to applications delivery that has tradi- 
tionally crippled many organizations in addressing the digital market opportu- 
nity. Rather than asking developers to build an application and then charging 
IT management with ongoing support, creating highly disparate and inefficient 
dynamics, DevOps brings those specialists together so that engineering is 
completed with a constant eye toward ongoing management. 


Just as critical in boosting organizational efficiency and software quality, 
DevOps methodology—like the Toyota Way—also promises to increase the 
ability to change the existing code base and deliver new capabilities to end 
users, while cultivating internal experimentation. Leveraging automation to do 
so is another hallmark of the Japanese carmaker's model and is also core to 
the DevOps approach. 


In a May 2014 editorial published in the Wall Street Journal, noted technology 
evangelist and The Phoenix Project co-author Gene Kim echoed the words of 
Deming in framing his view of ongoing DevOps adoption. Titled "Enterprise 
DevOps Adoption Isn't Mandatory — but Neither Is Survival,” Kim, also an 
established entrepreneur, posits that "the business value created by DevOps 
will be even larger than was created by the manufacturing revolution." 


™Enterprise DevOps Adoption Isn't Mandatory — but Neither Is Survival,” by Gene Kim, 
copyright WSJ, 2014: http://blogs.wsj.com/cio/2014/05/22/enterprise-devops- 
adoption-isnt-mandatory-but-neither-is-survival/ 
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Responding to some experts' observations that DevOps is only relevant for 
lean-minded, applications-driven startups such as consumer transportation 
darling Uber, Kim asserts that even the oldest, most entrenched businesses 
must wrap their arms around the movement to compete and survive. The 
author contends in the WS] piece that this is, “because IT is the factory floor 
of this century, and not just for manufacturing companies. IT is increasingly 
how all businesses acquire customers and deliver value to them.” 


Like the so-called "lean manufacturing" wave that swept the business world in 
the 1980s—directly related to the success of Toyota in growing to the point 
of global sales domination—Kim asserts that today's organizations must seek 
to drive every element of inefficiency out of their software development life- 
cycle (SDLC). 


Just as companies that failed to adopt leaner manufacturing processes in the 
latter half of the 20th Century lost out to rivals that did, the expert, among 
many others, maintains that organizations who overlook the need to adopt 
DevOps in today's Applications Economy will likely disappear. 


Evidence that Kim and other proponents are correct in predicting that orga- 
nizations’ willingness to embrace DevOps will directly impact their ability to 
compete is already mounting. Numerous research reports have reinforced 
that leading DevOps practitioners already appreciate significant competitive 
benefits. 


In a 2015 study published by Freeform Dynamics, in partnership with CA 
Technologies, researchers found that the 20 percent of organizations that had 
broadly adopted DevOps methodologies were 2.5 times more likely to have 
charted improvements in customer retention. 


The report, “Assembling the DevOps Jigsaw,” also found that these early 
adopters were 2 times more likely to have realized improvements in cus- 
tomer acquisition, and 3.4 times more likely to appreciate growth in mar- 
ket share. Perhaps most notably, those organizations already well into their 
DevOps transformations were 2 times more likely to have recorded a posi- 
tive impact on revenue, and 2.4 times more likely to have experienced higher 
profit growth. 


Regardless of the given era and environment, those financial results would 
seem to speak for themselves. Yet, according to the Freeform Dynamics sur- 
vey, 80 percent of all organizations have yet to embrace DevOps completely. 


8'Assembling the DevOps Jigsaw,” Freeform Dynamics, copyright 2015:http://rewrite. 
ca.com/us/articles/devops/assembling-the-devops-jigsaw.html 
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DevOps as a Critical Requirement 


The reality is, whether organizations are ready or not, the requirement to aggres- 
sively adopt DevOps methodology has quickly become the new normal in the 
Applications Economy. At the very least, the notion that such change is inevitable 
in cultivating continued business growth must be recognized as an inconvenient, 
yet irrefutable truth. 


In an increasingly fast-paced, complex, and ambiguous business world, compet- 
ing on a landscape dominated by the continued evolution of digital channels 
and mobile devices, the only tractable strategy is to adapt to survive. Driven by 
the exponential speed of change, organizations must wrap their arms around 
DevOps if they hope to defend and expand their market opportunities. 


Within this current atmosphere—defined by volatility, uncertainty, complex- 
ity, and ambiguity (VUCA)—DevOps offers an unprecedented opportunity 
for organizations to transform their SDLC to increase efficiency and meet 
end users' changing expectations. By fundamentally recasting the manner in 
which they approach every element of software development and manage- 
ment, DevOps represents the broader future of business in general. 


Despite the tendency to celebrate industry “unicorns” such as Uber—start- 
ups whose technology and business models immediately lent themselves 
to DevOps adoption—research such as the Freeform Dynamics “Jigsaw” 
report illustrates that organizations of all sizes and industries must change, 
or risk potential obsolescence. This conclusion is already being proven out 
by widespread assimilation of the involved methodologies by everyone from 
lean startups and centuries-old manufacturers, to nonprofits and government 
agencies. 


Banking on DevOps Practices 


Adoption among stalwart names in the banking industry—recognized as one 
of the most entrenched and "old school" segments in the entire business 
world—offers further proof of this pervasive need for DevOps transformation. 


While notoriously staid in some senses, banks have also long-embraced signifi- 
cant levels of software and IT to diversify their services and increase profit- 
ability. Examples include inventions such as automated teller machines (ATMs) 
and the vast electronic transaction processing systems that allow these com- 
panies and their clients to move capital around the world in real time. 


With roots dating back over 150+ years into the Dutch banking trade, 
global corporation ING is one such old world company that has success- 
fully embraced DevOps transformation. Driven by existing inefficiencies in 
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its 15,000-strong IT workforce and the need to address changing customer 
demands to increase its (EU) 15 billion annual revenues, the Amsterdam-based 
company embarked on an aggressive DevOps strategy beginning in 201 1.’ 


Aimed specifically at enabling so-called "continuous delivery" of its applica- 
tions to suit emerging customer preferences around online and mobile bank- 
ing, company officials sought to remodel SDLC methodologies by invoking a 
manifesto of legendary Apple co-founder Steve Jobs. Citing a 1989 interview 
with Inc. Magazine in which Jobs famously said, “You can't just ask customers 
what they want and then try to give that to them. By the time you get it built, 
they'll want something new," '? ING set about on its DevOps initiative. 


With the expressed goal of retrenching its SDLC to eliminate onerous, 
time-consuming waterfall practices and accelerate the pace of applica- 
tions innovation, ING first moved to create more agile “scrum” teams that 
brought together development and operations expertise. From a process 
standpoint, one of ING’s tactical goals was to begin testing new applica- 
tions code in a far more "production-like" environment, such that emerging 
issues could be identified and resolved faster, accelerating code delivery to 
end users, while improving quality. 


In support of those initiatives, the banking giant also leveraged significant 
automation to address SDLC requirements, including configuration manage- 
ment and software deployment. By bridging the gap between dev and ops, and 
employing new levels of automation, the company was able to increase the 
pace of its applications releases from an average of once every 13 weeks to 
weekly updates. 


In support of this larger continuous delivery effort, ING was also able to 
increase its pace of underlying mobile applications code deployments from 
several hundred per month to over 80,000 per month, in less than two years. 
By leveraging DevOps workflows and supporting automation, the Dutch bank 
transitioned from release cycles dictated by technical roadmaps, to those 
focused on strategic business objectives. 


Ultimately, customers expressed their approval both in adoption and via pub- 
lic forums, as ING's mobile application reviews on the Apple iTunes App Store 
climbed from one star in 2011, to four stars in 2014. 


?'|NG Bank Case Study,’ CA World’ 14 Presentation copyright CA Technologies.https : / / 
www. youtube.com/watch?v-9jqY bvI5vk 

!*The Entrepreneur of the Decade,” by George Gendron and Bo Burlingham, copyright 
Inc. Magazine 1989: http: //fortune.com/2009/11/23/decade-steve- jobs-apple/ 


— 10| 
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DevOps: A Key Component of Business Agility 


The ING story stands out not only as detailed proof of the manner in 
which DevOps represents a tremendous opportunity SDLC reinvention, but 
also as a specific example of how large, entrenched organizations can also 
emulate the much admired startup mentality. At the end of the day, the 
company’s continuous delivery exercise allowed it to appease the changing 
preferences of mobile applications users, the ultimate goal of countless new 
startup ventures. 


For years, everyone from management consultants to Wall Street inves- 
tors have beleaguered the need for such pre-Internet companies to better 
leverage the “lean and mean” traits of their upstart peers. With the dawn 
of the DevOps era, driven by the Applications Economy, organizations, 
regardless of scale and history, have in fact been presented with this spe- 
cific opportunity. 


By completely transforming the manner, and more importantly the speed 
with which new ideas can be translated into marketable products and ser- 
vices—in the specific form of applications—organizations that have tra- 
ditionally been slowed by their inherent size and complexity can rapidly 
accelerate innovation. 


Conventional wisdom dictates that large enterprises with sundry customers 
and products face greater risk in attempting to launch new services and sup- 
porting business models. Meanwhile startups, unburdened with existing pro- 
cesses and systems, can rapidly pivot from one guiding approach to another, 
reinventing themselves whenever the need arises. Enterprises, typically guided 
by the need to requite shareholder interests, have been prone to err more 
toward stability, limiting their rapid growth potential. 


For example, it is widely recognized that this entrenched, risk-averse men- 
tality has allowed startup companies such as eBay and Airbnb to displace 
longstanding giants of the global retail and hospitality industries, respectively. 
If you could turn back the clock 5-10 years and give Sears and Hilton another 
chance to adopt the same strategies used by their startup rivals, obviously 
they would jump at the opportunity. 


Part of this, in addition to a lack of shareholder reckoning, revolves around 
the fact that startups have also been able to quickly embrace emerging busi- 
ness technology trends, including cloud computing, mobile apps, open source, 
crowdfunding, and other means of collaborative economics. 


Yet, if the core of this mentality, empowered by technological agility, is that 
innovation is merely a “good idea made possible,” then adoption of DevOps 
represents just such an opportunity for organizations, regardless of size, to 
accelerate business via more efficient applications delivery. 
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As framed by Jez Humble, leading technology consultant, recognized as a 
founding father of the DevOps movement: “The long-term value of an enter- 
prise is not captured by the value of its products and intellectual property, but 
rather by its ability to continuously increase the value it provides to custom- 
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ers—and to create new customers—through innovation. 


At its core, the premium placed on the startup mentality, as previously refer- 
enced, relates primarily to the notion of market disruption. This is perhaps 
best exemplified, thus far, by those Applications Economy darlings that have 
successfully introduced new business models supported by web and mobile 
technologies. 


However, leveraging DevOps and its halo effect of increased innovation 
through more efficient and rapid delivery of differentiated applications, one 
could easily assert that in the matter of enterprise versus startup, the playing 
field is being leveled quickly. Specifically, as organizations leverage DevOps to 
become more agile in addressing the Applications Economy, they are able to 
better keep pace with their smaller, more innately nimble rivals. 


Circling back to the origins of the Toyota Way, the involved tenets of disrup- 
tion via technological innovation have always been recognized as catalyst in 
transforming industry; in fact, even the genesis of this specific example has 
roots predating the auto industry. 


In 1896, Sakichi Toyoda invented Japan’s first self-powered loom, incorporat- 
ing numerous revolutionary features, most notably the ability to automatically 
halt production when threads moving through the devices were broken."? 
Leveraging his invention, Toyoda built his startup into a leader in the Japanese 
textiles industry. 


Those concepts, which became the legendary Toyota Way, were merely car- 
ried over when his son Kiichiro launched the automaker Toyota Motors in 
1937. As it would turn out, this technology-driven approach to innovation and 
disruption would also dovetail perfectly with the lean management concepts 
promoted by Deming as he helped resurrect Japanese manufacturing in the 
1940s. 


At the time, automakers in other areas of the globe, notably the United States, 
likely would have never accepted that such a company, far smaller than their 
own booming operations and literally rising from the ashes of war, would have 
the opportunity to dominate the worldwide market. 


"Jez Humble, Lean Enterprise: How High Performance Organizations Innovate at Scale, 
(Sebastopol, California, O'Reilly, 2015) p. 39. 

"" Automation with a Human Touch,” Toyota Motor Corporation World site, copyright 201 6: 
http://www. toyota-global.com/company/vision_philosophy/toyota_production_ 
system/jidoka. html 
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In 2012, after decades of displacing customers from other brands, Toyota was 
recognized at the world's largest automobile manufacturer, having produced 
its 200-millionth vehicle and grown into the first automobile manufacturer 
to ever produce more than 10 million vehicles per year. In 2016, amid larger 
worldwide economic concerns and a headline-grabbing airbag recall issue, the 
company remains among the global sales leaders in its industry. 


Companies such as Toyota and ING, along with endless agile startups, offer 
tacit proof that for organizations seeking to increase market relevance and dis- 
ruption, DevOps offers an attractive template. Yet, citing Freeform Dynamics’ 
findings that only 20 percent of all organizations have achieved any semblance 
of DevOps maturity, the movement is clearly still in its nascence. 


To wit, the researchers found that even though 80 percent of respondents to 
its survey view DevOps as a “key component of business agility,’ only 55 per- 
cent of organizations laid claim to having created a well-defined DevOps strat- 
egy. Furthermore, while 86 percent of those organizations surveyed labeled 
DevOps-oriented education of business stakeholders and greater alignment 
of IT and business priorities as important, only 33 percent and 37 percent, had 
enacted those programs, respectively. 


Loosely stated, the concept of disruption, in general, suggests that market 
incumbents typically become less profitable and open to displacement from 
more agile peers based on the entrenchment of inflexible business models. 
It is widely acknowledged that there are two primary models for market 
disruption in the current era—‘new market” disruption and “low end" dis- 
ruption—at least as defined within the context of so-called “disruptive inno- 
vation” theory." 


In the case of new market disruption, the landscape is forever altered when a 
new products arrives that somehow better addresses a segment that incum- 
bents have not yet envisioned or delivered. Whereas with low end disruption, 
incumbents see business chipped away by products that aim to encompass a 
smaller subset of perceived value drivers offered at a lower price point. 


Looking closer at the promise and outfalls of DevOps—and the ability of 
adopters to better align themselves with the exploding demand for software 
and applications—one could easily argue that this ongoing technological rev- 
olution directly supports both archetypes of the disruptive opportunity. It 
would seem obvious that those organizations that have yet to jump in sit 
greatly imperiled by its continued advancement. 


Some may view the Darwinian views of Deming, or Gene Kim, as overstated, 
particularly as related to DevOps and the Applications Economy. One could 
imagine that organizations that do not yet supply applications to consumers 
or business partners would remove themselves from the larger conversation. 


'WWhat is Disruptive Innovation?,” copyright Harvard Business Review, Dec. 2015. 
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However, in the following chapters of this book, there is a great deal of evi- 
dence—backed by specific technical practices—that make it clear just how 
sensible, if not necessary, it is for every organization to view the DevOps 
opportunity as their best chance for success and survival. 


DevOps: A Practice for Champions 


At the conclusion of the 2015 FI season, the world champion driver Lewis 
Hamilton and teammate Nico Rosberg of the Mercedes AMG Petronas 
Formula | Team, also winners of the highly coveted FI constructor's title, 
returned to the outfit's UK headquarters to thank the factory workers who 
made their victories possible. 


While most of those people, numbering in the hundreds, never joined the 
team at the track on a single race day during the globe-spanning nine month 
season, the champions heaped praise on their colleagues, recognizing the year- 
round effort that supports the entire operation. 


Mercedes made easy work of the rest of the FI field in 2105, having won a 
stunning |6 of the 19 total races. At the same time, this was not without 
constant updates to its racecars, with new components constantly meeting 
the team around the world as they progressed throughout the grueling race 
calendar. 


“In racing there are always things you can learn, every single day. There is 
always space for improvement, and | think that applies to everything in life,” 
Hamilton has been quoted as saying.'* 


"With the team, whether it be my guys that are at each Grand Prix, the team 
back at the factory who work day and night to develop the car, build the parts, 
and send the parts, the IT team, and the crew cleaning the factory. Everyone. 
We do it together. We all feel the joy when we are winning and the same pain 
when we are losing,” the champion driver continued. 


Without the constant delivery of new innovations, without the continued 
experimentation and refinement of the involved technologies, and of course 
all the underlying processes, Hamilton affirms, the likelihood of his success 
would be scant, if not impossible. 


DevOps offers the chance for any organization to run more like a champion- 
ship team, ever improving performance, going faster, and winning. 


"Mercedes AMG Petronas Team End Season on Top,” By Donny Halliwell, Inside Blackberry, 
copyright2015.http: //crackberry.com/mercedes-amg-petronas-formula-one-team 
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Summary 


Only a few years into the Application Economy it's clear that huge benefits 
are being appreciated by those organizations capable of embracing emerg- 
ing processes and technologies that enable agile methodologies. DevOps is 
widely recognized as one of the most influential and important movements 
that empower that transformation. 


At the same time, such a sea change in the manner that organizations approach 
software development and operation, or any process for that manner, cannot 
be realized without some growing pains. In the next chapter, we'll take a look 
at the historic state of disconnect that has existed between developers and IT 
operations staff, and the approach that organizations embracing DevOps have 
employed to reinvent the manner that such experts collaborate. 


CHAPTER 


2 


IT Impasse 


Faster Software Development Versus Operational 
Stability 


Ever since we flipped the switch on commercial computers back in the 1950s, 
IT departments have been struggling to keep up with an insatiable demand 
for software applications and services. Of course many technologies like 
commercial of-the-shelf software packages, virtualization, and cloud comput- 
ing have helped along the way, but generally IT delivery has been slow and 
uncoordinated. 


In such an environment where IT generally supported internally-centric busi- 
ness processes, separate teams overseeing discrete technology functions was 
for many years considered the norm. But in today's rapidly evolving digital 
world, these practices no longer work. 


A World of 'Wicked' Business Problems 


Keeping up with demand for changes to internal applications supporting 
business processes is tame compared to the "wicked" problems facing IT 
departments today. Like societal problems such as global warming and drug 
abuse, they're wicked because IT is placed in an unenviable position of trying 


© CA 2016 
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to rapidly deliver solutions before problems are fully understood and where 
business conditions constantly change. This is due to three transformational 
forces: 


e Products to services—Customers now care less about 
physical things and more about the total experience. This 
explains why companies wrap physical products in ser- 
vices. Like Tesla, who routinely deliver enhancements to 
the Model S car via software. Or Bosch, who now provides 
tailored telematics and analytics as-a-services so that fleet 
operators can optimize maintenance and cut fuel costs. 


e Efficiency to agility—Established brands with decades of 
reputation are constantly being disrupted. Kodak lasted 
100 years, Blockbuster less than 20. Even technology stal- 
warts like Microsoft have felt the ground shift beneath 
them. Business reputations are forged from digital adept- 
ness and the ability stay ahead of the market—or create 
new ones. 


e Separation to fusion—There is no longer separation 
between physical products and software. Is your smart- 
phone circuitry and plastic, or is it a Spotify music service 
and digital payment system? Is your Nest home thermo- 
stat an aesthetically pleasing appliance, or is it an analyti- 
cal marvel of energy management? 


Operating within this environment, wicked problems now challenge the 
essence of how business is conducted and how applications should be 
designed, developed, and supported. Thanks to mobility and social comput- 
ing, the producer-consumer relationship has been reversed, with businesses 
placed in a position of having to respond to customer behaviors rather than 
dictating them. This places many organizations on the back foot because of 
traditional development practices. 


Internal business processes supported by large complex applications have 
been the lifeblood of business. Supporting processes like logistics, inventory, 
and financial management, these applications are periodically optimized to eke 
out operational cost efficiencies. In this context, the focus on IT has been to 
maintain a steady state; keeping the technical lights and only changing applica- 
tions over longer cycles, sometimes only once or twice a year. 


Even when custom development occurs, the general practice has been to 
invest heavily in application software to support large-scale projects. Here, 
the pattern is to define all the requirements and features at the start and 
progress the application toward production status by developing and testing 
in a linear fashion. Finally, if many business stakeholders agree the system is 
what's wanted, the application is handed over the wall to production for IT 
operations to maintain. 
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Considering these forces, this “Waterfall” style of development (see Figure 2-1) 
now has a number of disadvantages. First, by establishing a fully defined set 
of requirements up-front and then failing to accommodate shifting customer 
behaviors, organizations risk delivering "white elephant" applications, which 
are software systems that customers never use. Secondly, in the time taken 
to release software, business conditions may have changed, meaning depart- 
ments must change the system radically, or as is often the case, abandon it 
completely. Finally, since application software and functionality is delivered 
en masse, the support and maintenance burden on IT operations increases 
suddenly and significantly. 


User Requirements 


Aimingto fully define requirements at 
the start and excessive time delays 
Business Case results in services that don't meet 
/ Funding business needs and burden IT with 
more cost. 


Problem 


Development 


Testing / QA 


Implementation 


Problem ——————> 


Solution 


Time 


Figure 2-1. Waterfall software development model 


The Emergence of Agile Development 


In Formula | racing, rules change every season. Cars now compete without 
fuel pit-stops and on sets of tires that degrade more quickly. It's the same in IT, 
where software must now be delivered as a continuous stream of value that 
constantly changes to support new business conditions. 


Trying to solve wicked business problems (e.g., transforming the businesses 
operational model with a new mobile sales channel) with existing methodolo- 
gies like Waterfall can be like trying to corral cats. Problem definitions and 
requirements can change as new solutions are considered and implemented, 
so much more flexibility is required across the software development lifecycle. 
And, with many diverse opinions on what actually constitutes a business prob- 
lem, there'll be many stakeholders to engage and ideas to analyze. 


17 


— 18| 


Chapter 2 | IT Impasse 


User Requirements Planning 


Development 
“Sprints” 


Operational Feedback Retrospectives iterations... 


Deployment User Review 


Figure 2-2. Agile software development model 


Agile development has emerged as the practice of choice to address these 
challenges. Agile (see Figure 2-2), which involves iterative problem solving and 
continuous feedback, is well suited to solving these complex business prob- 
lems. Some common benefits include: 


e Software updates are made in small regular batches, 
allowing businesses to adapt quickly to new information 


e Mistakes or false assumptions are detected much earlier 
in the software lifecycle, where they are much less costly 
to correct 


e By detecting problems quickly and early, teams have much 
faster feedback and organizational knowledge improves 


e Teams can immediately apply learning and knowledge 
gained to improve the quality of application software 


e By working in parallel across development and testing, 
agile teams can increase the velocity of application devel- 
opment and delivery 


Agile methods help teams manage and run their own projects. Rather than 
wait for approvals and sign-offs across multiple stages, agile supports the 
notion that fully autonomous teams should be able to make their own archi- 
tectural design decisions, even when it comes to tool usage and deciding 
what's needed to push software code all the way into production. 
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But agile development is not without its drawbacks. While agile development 
teams are focused on speed and agility, the traditional mantra of IT operations 
is maintaining application stability, even if that means slowing things down. To 
this end, IT operations often employ rigorous change control processes and 
enforce infrastructure standardization dictates to maintain control. While well 
suited to supporting legacy internal systems where updates were delivered in 
large batches, these processes can be too rigid to support newer applications 
designed to be always changing. 


Many of these issues are a direct consequence of how IT operations have 
traditionally been run. Historically, [T operations have been delivered as a set 
of shared services, supported by functional disciplines in areas such as data 
center services, network management, and security. If business users or devel- 
opment needed an enabling service (e.g., provisioning test labs), operations 
had to be engaged for delivery. 


However, this model is changing, with new tools and methods that enable 
business groups and development teams to bypass the IT operations shared 
service completely. Many recent technology trends have supported these 
practices, including: 


e Polyglot programming languages/platforms—Autonomous 
teams can select the programming language best suited 
to their project. This may include older languages like 
Java and C++, or new platforms and languages to support 
specific development use-cases or design requirements 
(e.g., Node.JS, Go, and Rust). In other cases development 
teams are eschewing traditional relational databases in 
favor of NoSQL or document style data stores. These 
could be advantageous in projects where data require- 
ments are indeterminate or evolving and where speed 
and scalability is more critical than up-front logical 
design and data integrity. Examples include MongoDB, 
PostgreSQL, and Cassandra. 


e Programmable Infrastructure—Also known as Infrastructure- 
as-code, this allows teams to write code to manage 
configurations and automate infrastructure provisioning. 
Rather than using manual methods or scripting to build 
development ready infrastructure, teams can write code in 
languages with which they are familiar, together with famil- 
iar practices such as version control and automated testing 
to reduce errors. Unlike scripting, which is primarily used to 
automate static steps, programmable infrastructure with its 
descriptive languages allows developers to code processes 
that are far more versatile and adaptive. Examples include 
Chef, Puppet, and Ansible. 
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Open source software—Beyond the more obvious "free 
to use" benefits, many organizations are embracing open 
source software (OSS) as a means to increase agility. 
By adopting OSS, development teams can benefit from 
community-based and collaborative development, with 
tools that simplify and automate many tasks (e.g., version 
control and software build management). By having the 
eyes of the "community" on the software, bugs can be 
quickly identified and resolved, with new improvements 
constantly being developed and delivered. 


Cloud and platform-as-a-service (PaaS)—Not too long ago 
testing a new application or update required develop- 
ment managers to submit a business case for the pro- 
curement of new hardware, wait weeks for delivery, then 
go through a painstaking process of provisioning and con- 
figuration. This has changed in recent years—first with 
the advent of server virtualization and more recently 
with PaaS providing complete cloud technology stacks 
for development and testing. 


Containers—A recent technology innovation, containers 
comprise an entire runtime environment, including the 
application, plus all its dependencies, libraries, and con- 
figuration files needed to run it—all bundled into one 
package. By containerizing the application platform and 
all dependencies, any differences in OS distributions and 
underlying infrastructure are abstracted away. Unlike 
virtual machines, containerized applications run a single 
operating system, and each container shares the operat- 
ing system kernel with the other containers. This makes 
containers more lightweight and resource efficient than 
virtual machines. Containers can be run on public cloud 
services and are easily shared, which makes them particu- 
larly useful for development and testing teams. 


Canary releases/dark launches—This is a release method 
used to test the performance of software deployments 
and new functionality in a phased manner. Canary releases 
are an important aspect of agile development, since, by 
rolling out new features incrementally and testing them 
with real users, teams can gather feedback and implement 
improvements quickly. 
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e Design for failure—As businesses embrace public cloud 
services, teams are employing design methods to make 
applications completely independent of the availability of 
underlying infrastructure. By building resilient systems in 
which inevitable failures have a minimum impact on ser- 
vice, design for failure allows teams to scale applications 
quickly while achieving higher levels of uptime, even dur- 
ing major cloud infrastructure outages. 


Agile Empowerment Challenges 


While many of the technologies and methods described above have empow- 
ered teams to deliver software faster, this doesn't guarantee success. While 
developers are tasked with producing great software code, they often neglect 
critical issues associated with application performance and supportability. In 
many cases the adoption of new technologies and practices exacerbates this 
problem due to some common failings: 


Technology for Technologies’ sake—lt's understandable that developers want to 
use the latest tool, but this can lead to support problems. For example, opting 
for a NoSQL database because it helps avoid lengthy schema design issues 
may address short-term issues, but increases the support burden because IT 
operations and support have no experience with the technology. Operational 
cost structures can also be impacted, by way of increased training costs and 
hiring specialists. 


Tip Make new technology tool selection a collaborative exercise. Enterprise architects can 
coach teams on the importance of placing program or business level objectives above discrete tool 
requirement or personal preferences. 


Misuse of New Technology—lt’s a fact of IT that new technologies often can 
and will be abused, unwittingly or intentionally. With modern technologies, 
lack of experience may entice people to use technologies in ways they were 
never designed. For example, blindly dictating that every monolithic and legacy 
application should be containerized whether they're appropriate for the tech- 
nology or not. 


Tip Today's new technology is tomorrow's legacy. Always assess whether the use of technology 
is appropriate for the business and never underestimate architecture and design requirements. 
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Metric Misalignment—With developers using techniques like canary releases, 
dark launches, and split or A/B testing, businesses needs to know whether 
new features are successful and have led to more customer conversions and 
sales. Historically, however, IT operations have used technical diagnostics as 
a means to assess performance, which may be misaligned with the business. 


Tip Consider adopting monitoring methods that focus more on achieving desired business 
outcomes. In a mobile app context, this could include monitoring techniques to analyze app and 
functional usage by geographic area and user community. By managing to these outcomes, it 
becomes easier to assess the implications of any application performance issues and feed richer 
information and insights back to development. 


Lost Opportunities—As new development platforms become increasingly 
abstracted from hardware infrastructure, many complex performance prob- 
lems can surface. Because developers are shielded from the infrastructure, 
they can be slow to respond to any issues their code has introduced. This lack 
of insight also means they can fail to fully exploit the true potential of new 
cloud-based technologies. 


Tip In customer-centric computing, high-performance and low-latency are as important 
as functionality and design. Consider teaming junior and experienced developers with skilled 
IT operations and sysadmins to determine how modern hardware architectures can be better 
exploited to consistently deliver a high-quality customer experience. 


Workarounds Due to Constraints—Even as teams look to adopt public cloud ser- 
vices and PaaS to accelerate development, they're still constrained by access to 
production systems for testing. This can involve delayed access to applications, 
infrastructure, and perhaps the most time-consuming and resource-intensive 
task in [T—creating, maintaining, and provisioning accurate, compliant and up- 
to-date production-like test data. With these constraints, teams may introduce 
sub-optimal testing practices that fail to detect software defects or compro- 
mise compliance with regulatory data protection requirements. 


Tip Consider technologies that can simulate unavailable systems across the development 
lifecycle. This allows developers, testers, and integration teams to work in parallel for faster 
delivery. Capabilities in test data management, including data subsetting and masking, together 
with synthetic on-demand test data generation, should also be assessed. 
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Modern Application Architectures 


To support the businesses need to innovate, development teams must contin- 
uously deliver software services at an increased velocity. This has been recog- 
nized by web-native companies such as Netflix and Amazon, who've changed 
their software architectures to support the need for continuous innovation— 
essentially using them to redefine the markets within which they operate. 


With agile methods, organizations can iterate quickly to support innovation, 
but teams are also changing application architectures to improve software 
flexibility and help accelerate deployment. For this reason, older style monolith 
designs are being supplemented with microservice designs (see Figure 2-3). 


Monolith Architecture Microservice Architecture 


All application functionality in a single process Each element of functionality in a separate service. ... 


User Interface 
User Interface 


Cross-functional teams organized 
around business capabilities 


Business Logic 
Microservice Microservice Microservice 


Data Access 


Silo'd functional Database Database Database 
teams 


Database 


Scales by distributing entire application Scales by distributing services, replicating as needed 


Figure 2-3. Monolthic and microservice application architectures 


Until recently, application architectures were monolithic in design and opera- 
tion. Although consisting of many services, monolithic applications are pack- 
aged and operate as a single unit. For IT teams now tasked with faster delivery 
and deployment, the characteristics of monolithic design have presented a 
number of operational challenges: 


e  Brittleness—lf any single application element of compo- 
nent fails, then the entire application fails. If a task such 
as payment processing consumes more CPU or memory, 
then the whole application can degrade. 


e  Risk—Because everything is packaged together (and fails 
together), teams may be hesitant to change supporting 
technology stacks and infrastructure. This can explain 
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Microservice designs differ from monoliths in that they involve building appli- 
cations as a set of small, independent services. In essence, each microservice 
focuses on a specific element of business functionality. For example, in a web 
application there could be many microservices supporting everything from 
login processing and shopping carts, recommendation services and payment 
processing. Loosely couple in nature, microservices communicate with other 


operational resistance to increased deployment rates, 
since even simple application updates to brittle mono- 
lithic applications could cause system-level outages. 


Tightly coupled—Applications can only be scaled by 
deploying the entire application on more servers. This can 
be highly problematic in new mobile application develop- 
ment scenarios when demand is difficult to predict. 


Dependencies—Since applications are tied together, 
developers may have difficulty working independently to 
develop, test, and deploy their own software components. 
Development time increases and productivity suffers 
because they're often dependent on other teams finish- 
ing work before they can start. 


services via Application Programming Interfaces (APls). 


Microservices can provide business a number of significant benefits: 


Independent deployment—Scaling becomes less problem- 
atic. For example, in a web-based shopping application a 
team can quickly deploy the instances of service it needs 
to meet demand spikes, while scaling back others. 


Independent coding—Teams have more freedom to 
develop in different programming languages, each opti- 
mized for different processing tasks. Microservices can 
free organizations from being locked into a single tech- 
nology stack. 


Fault tolerance—VWVhen a microservice fails, it’s unlikely 
that the entire system will fail. If the recommendation 
service fails in a web application, shopping cart and pay- 
ment processing services can continue to function. 


Increased agility—Microservices designs can better sup- 
port continuous delivery. Since systems are built and 
deployed independently, they can potentially be tested 
and delivered faster. 
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Microservices: Small Isn't Always Beautiful 


While the simple and elegant nature of microservice style design delivers 
many benefits, substantial complexity exists in terms of management and 
operations. If ignored, these issues could increase friction between develop- 
ment and operations teams. Important considerations include: 


e  Supportability—The support burden can increase substan- 
tially—especially when IT operations are suddenly faced 
with managing hundreds of microservices developed in 
different languages, accessing new data stores, and run- 
ning on cloud platforms and infrastructure. 


e  Monitoring—Managing a single monolithic application is 
demanding, but now IT operations has to ensure many 
more processes remain performant. With highly distrib- 
uted microservices systems, a whole new set of man- 
agement considerations surface. These include network 
latency, fault tolerance, asynchronous messaging issues, 
and network reliability. 


e  Coordination—Deploying hundreds of microservices 
demands rigorous deployment processes that can be 
beyond the capabilities of teams using manual processes 
and scripting. 


Note Modern management approaches to address microservice deployment and monitoring 
challenges are discussed further in Chapters 6 and 7. 


Ending the Technical Impasse 


Even with advances in tooling, development methods and software designs, 
the friction created by teams operating in discrete functional silos can negate 
all potential benefits. Fractured processes across the application software 
lifecycle inevitably result in slow delivery, low productivity, and defect-ridden 
software systems. This has to change. 


IT's value proposition isn't just to keep the technology lights on and periodi- 
cally deliver improvements over long cycles; it's to continuously manufacture 
business value from a modern high-performance software factory. 


DevOps, with its focus on close collaboration between development and 
other IT groups, while automating essential application delivery processes, is 
now a critical business imperative. 
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Summary 


The following chapters outline key strategies that can help IT and digital lead- 
ers accelerate a successful and business-aligned DevOps initiative. 


Starting with Chapter 3, we'll describe how to build a winning DevOps culture 
and re-energize the IT organization. Here, we'll examine easy and cost-effec- 
tive ways to increase collaboration, the application of Lean thinking to reduce 
IT waste, and what constitutes a comprehensive DevOps metrics program. 


CHAPTER 


3 


DevOps 
Foundations 


Culture, Lean Thinking, Metrics 


Blink during a Formula | pit-stop and you'll probably miss it. But this wasn't always 
the case. Fifty years ago, a pit-crew would take over a minute to change the wheels 
and refuel. Today, anything more than three seconds is considered a fail. 


It’s the same in software development, where teams once tasked with updat- 
ing enterprise applications at a sedate pace must now deliver new software 
services as a continuous flow of value to customers. 


The problem for today’s enterprise, however, is that software teams don’t 
work like Formula | pit-crews. Rather than working in tandem, IT teams often 
work serially—development codes, then QA tests, and finally IT operations 
monitors. However, with application software released, enhanced, and retired 
over more compressed timeframes (months and even days), this stop-start 
method of development falls short. It’s as ineffective as each member of a 
Formula | pit-crew replacing a tire and checking wheel nut tension before 
the next one could start—the race would be over before the car left the pits. 


While we can celebrate the heroics and skill of great racing car drivers, what sets 
successful constructors apart is their ability to build a winning culture irrespec- 
tive of role and responsibility, be that driver, team manager, telemetry engineer, or 
aerodynamics chief, everyone is focused on a singular goal—winning races. It’s why 
drivers thank the teams before they spray champagne on the podium. 


© CA 2016 
A. Ravichandran et al., DevOps for Digital Leaders, DOI 10.1007/978-1-4842-1842-6 3 
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Like Formula | drivers, technological advancements have improved the effi- 
ciency and effectiveness of IT professionals. However, in organizations that 
traditionally measure and incentivize based on technical specialization within 
functional areas, relying on tools alone will never build the collaborative cul- 
ture needed for business growth and profitability. 


What Characterizes DevOps Culture? 


DevOps is very different from traditional thinking because it places great 
emphasis on culture. It instills a shared sense of vision across multiple teams, 
directly aligned to the business and its customers. To this end, maverick 
behavior, such a cutting corners and allowing defect ridden code to go into 
production, or blaming operations when a software release fails, is counter to 
a DevOps thinking. With DevOps unified IT is the hero and no one is singu- 
larly to blame for problems. 


But this is challenging in IT because of the friction existing between develop- 
ment and other IT teams—especially IT operations. On the one hand, devel- 
opers are focused on accelerating change by faster delivery of applications, 
while the operational mantra has been resilience and stability at all costs, even 
if that means holding back change. 


Evidence suggests, however, that while both these goals are equally important, 
they are not mutually exclusive. For example, the 2016 Puppet Labs "State of 
DevOps" report illustrated that high-performing IT organizations are well 
able to achieve faster software delivery along with increased resilience and 
stability. Clearly, DevOps high performers have ended the divisional “turf 
wars" by enacting strategies to re-shape entrenched silo thinking and behav- 
iors into a more powerful collective force. 


Since DevOps culture involves creating new shared values and behaviors 
across IT teams, leadership must play an active role in driving these character- 
istics across the entire organization. 


Focusing on Products over Politics 


Traditionally, IT teams have been organized in technical silos. Interaction and 
communication has been conducted through overly engineered and rigid 
processes. Software changes run the gauntlet of lengthy change-management 
processes, human intervention, and change review boards. Though not wrong 
per se, these elements were designed to cater to situations where change 
was less frequent but occurred in greater volumes, requiring more rigor to 
ensure operational stability and compliance. 


'https://puppet.com/blog/2016-state-of-devops-survey-here 
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A strong DevOps culture, however, is characterized by systems thinking. That 
is, a collective emphasis on service as a whole, not on discreet functional 
elements or processes. Rather than persist with technical fiefdoms, DevOps 
aims to break down barriers—organizing by product over structure and con- 
tinuously driving improvements in context of a products lifecycle, from the 
inception of an idea to full production status. Strong leaders recognize this by 
promoting open communication, using shared metrics, and establishing (even 
automating) feedback mechanisms within and across teams. 


Building Trust and Respect 


Over many years, respect has been garnered by individual contributors. Be 
that superhuman developers who crank out code, or on-call operations staff 
who fix problems at 4:00am. In a thriving DevOps culture, hero worship- 
ping takes a back seat to collective respect. With DevOps, everyone should 
respect the contributions of others and no one should be afraid of speaking 
up for fear of abuse and vilification. 


This is critical, because from healthcare to aerospace, studies have shown that 
bad practices and behaviors can over time become accepted as normal prac- 
tices—often with disastrous consequences (see Chapter 8 for further discus- 
sion on strategies to combat normalized bad practices). In IT this happens all 
the time due to power games and lack of respect. Even if new staff witness 
blatantly suboptimal practices, they'll be loath to report it for fear of rebuke 
and retribution by managers, eventually accepting the situation and practicing 
it themselves. DevOps leaders should be mindful of this and work across the 
product lifecycle to identify situations where violations are tolerated because 
people are afraid to speak up or look mean. 


Trust also plays an important role in DevOps culture. Just as the Formula | 
driver trusts his pit-crew to fit four wheels securely, so must cross-functional 
trust be established across IT. For development, this means trusting that the 
production performance information from operations can actually help in 
software refactoring and reducing technical debt. For operations, it means 
trusting new application design patterns will help the business scale. Everyone 
from security to enterprise architecture is part of the trust equation, and as 
the speed of software delivery accelerates, no DevOps program will be suc- 
cessful without it. 


Increase Empathy Everywhere 


It’s well understood how important the role of empathy plays in today's app- 
centric software design. Without understanding the emotional and physical 
needs of customers, together with their behavioral patterns, businesses risk 
substantial losses from their software investments. This explains why many 
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organizations conduct rigorous design experiments before any full software 
release. This is illustrated in the extreme by Google's “50 Shades of Blue" user 
interface testing exercise.’ 


Yet despite this, empathy is lacking within many enterprise IT departments. 
Teams usually operate in separate locations, so development and operations 
teams have few face-to-face opportunities necessary to share each other's 
pains, surface concerns, or raise issues. 


There are many simple but effective strategies leaders can use to build empa- 
thy. Not the least this should include building closer ties between develop- 
ment and support. Even with the greatest software and delivery processes it's 
important to understand their perspective and what they experience when 
dealing with customers. 


Tip When developing products or new features, put yourself in the position of the customer 
and support staff by examining all the situations where they might need help. 


Staff (including developers) should understand the importance of enabling 
a great customer experience. To that end, consider working directly with 
customers directly in the field in a variety of adverse situations. What hap- 
pens when as a customer you're trying to board an airplane and the scanner 
breaks? Or what's the impact when a mobile app crashes or bad network 
coverage means you can't call road side assistance? 


Obviously it's not always practical or feasible for developers to work this closely 
with clients; however, with analytics tools, staff can put themselves in the "shoes" 
of the customers and gain realistic insights into the customer experience. 


Open Communication Channels 


In 1968 a computer programmer, Melvin Conway, postulated that organizations 
that design systems are constrained to produce designs which are copies of 
the communication structures of these organizations.’ In a DevOps context, 
what's now dubbed Conway's Law has great relevance, especially in situations 
where organizational structures and closed communication channels prevent 
developers and operations from agreeing on IT performance objectives (e.g., 
increased change frequency and improved reliability). In such cases, it's pos- 
sible for team-based activities to be prioritized over more important cross- 
functional improvement strategies. 


*https://www. theguardian.com/technology/2014/feb/05/why -google- 
engineers-designers 
3Melvin E. Conway, "How do Committees Invent?," Datamation 14 (5) (April 1968): 28-31. 
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There are many possible solutions to this problem. The technology teams 
at Netflix and Amazon structure themselves around small teams, with each 
one responsible for a small part of an overall system. Spotify promotes col- 
laboration across team boundaries via agile development squads, chapters, 
and guilds, with a separate IT operations team providing all teams the support 
needed to release software themselves. 


Looking beyond technology-centric companies, there are other examples of 
businesses thriving because they've modified communication structures. Take 
Zara for example. 


Year after year in the fickle world of retail fashion and apparel, Zara continues 
to increase revenue and profit. For Zara, flexible responsiveness to customer 
demand, backed by a tightly integrated supply chain, is fueled by teamwork 
and collaboration. In retail stores, managers use real-time intelligence to 
place orders and feedback information directly to the point of manufacture. 
Different teams (including design, product management, and merchandising) 
use shared spaces and work closely together. With increased emphasis on 
communication from initial garment design to distribution to the shop floor, 
Zara has shortened product lead times and doesn't unnecessarily commit 
large volumes of product in advance of a fashion season. 


Considering approaches like these, cross-functional IT collaboration could be 
improved when: 


e Leaders apportion budget to practical co-location strate- 
gies. This should be more than simple staff re-housing 
and include shared work spaces and lounges, together 
with team huddle areas and common wall whiteboards. 


e |T operations team members regularly participate in agile 
standup meetings in order to appreciate the value of 
deploying code quickly and how their current activities 
help or hinder release processes. 


e Development teams attend operational post-mortems 
or workshops to gain better insights into the problems 
caused by poorly performing or insecure software. 


e IT operations works with developers to establish perfor- 
mance monitoring in pre-production so as to detect prob- 
lems earlier where they are easier and less costly to fix. 


e Developers are placed on the after-hours support roster 
to better appreciate the impact of problematic software 
code on users and customers. 


e Support specialists share critical application experience 
analytics obtained during mobile app engagements with 
customers. 
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Additional Factors 


Changing IT culture isn't easy. Right or wrong, people have pre-conceived 
notions and firmly entrenched ideas. Any sudden shift in workplace practices 
and it's only natural that people will feel threatened and push back. Addressing 
this means patiently working with people to enact the necessary behavioral 
change or move people to different jobs. 


With DevOps and cultural change, it's critical to start with a clean state. 
Rather than dive head first into massive IT workforce transformation pro- 
grams, leaders should first assess the cultural landscape from a business per- 
spective. This involves understanding the primary goal of the business and 
then analyzing whether prevailing behaviors support it. Although seemingly 
obvious, many organizations miss or neglect this critical step. If, for example, 
a company defines its goals too broadly, people working in different IT teams 
will interpret them in different ways and shape activities accordingly, often to 
the detriment of each other and the business. 


In a broader sense, culture will also be influenced by an organizations' business 
model and operational perspective. There are three classes to consider: 


Run the business—ln this model the overarching strat- 
egy of the business is on continuing operations in much 
the same way—only better, faster, and cheaper. Here the 
focus for IT is operational excellence—adopting new 
technologies, yes, but using them to pound out efficien- 
cies across processes like warehousing and logistics. 


For these organizations, IT culture is characterized by 
discipline and rigidity—all fine to support efficiency 
improvements, but inadequate in a world where old 
business rules are constantly being challenged by dis- 
ruptive technology. 


Grow the business—The business strategy shifts from 
doing more of the same to conducting the same busi- 
ness in radically different ways. Netflix presents a good 
example of this model. Five years ago, Netflix delivered 
DVDs through the mail, now they stream entertain- 
ment over the web—even creating their own content. 
Customers still turn to Netflix for entertainment, but 
the way in which Netflix is addressing that need has 
fundamentally changed. 


For organizations in this category, the culture also 
needs to change. For Netflix, their DVD-delivery model 
required strong operational oversight over processes 
like inventory management and distribution to drive 
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efficiencies and increase customer satisfaction. Now 
however, their streaming model and content genera- 
tion programs requires teams rapidly delivering new 
services based on quickly analyzing customer prefer- 
ences and optimizing web performance and through- 
put. Obviously if a company's cultural behaviors and 
values are still skewed toward driving efficiency in one 
operational area, pivoting to the new model will be 
difficult. 


Transform the business—This model carries the most 
promise and risk because it involves changing the very 
fabric of a company. Businesses don't just conduct 
the same types of business in new ways, they rein- 
vent themselves completely. For Amazon that's meant 
moving to being a mega cloud computing provider 
from selling books. For Walgreens, it's meant going 
from selling medicine over the counter to treating ill- 
nesses in stores. 


With strategic transformation examples like these, 
the business is introduced to new dynamics and com- 
petitors. Now, IT performance will not only be judged 
on growing the business of today, but also on creating 
the core business in the future. To this end, a DevOps 
culture built on open communication and collabora- 
tion, trust, respect, and empathy isn't just important 
for short-term growth, it's essential for long-term 
business sustainability. 


Once business models and goals are clearly understood and communicated, 
any required [T behaviors and values needed to support them can be influ- 
enced through the development of fully aligned IT goals and metrics. These 
could include shortening lead times for new products to support faster time- 
to-market, increasing mobile app customer conversions to support increased 
revenue objectives, or helping the current business scale by making better use 
of cloud infrastructure. 


Lean Thinking to Reduce Waste 


Fuel strategies play a significant role in Formula | racing. A car with a half-full 
tank can be as much as three seconds faster than a rival vehicle with a fully 
loaded fuel cell. Extra fuel equals extra weight, so teams go to great lengths to 
calculate the exact amount of fuel needed under full race conditions. 
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Though beautifully engineered and continuously refined, racing car engines 
are still flagrantly wasteful. Like your car at home, the dynamics of internal 
combustion still mean that only a certain percentage of fuel stored in the 
tank is converted into useful energy. The rest is lost as heat and friction 
and explains why teams constantly refine chassis designs to reduce aero- 
dynamic drag. 


Beyond the frenzied world of Formula |, many elements contribute toward 
engine waste in the cars we lesser mortals drive. Running an air conditioner 
consumes fuel without contributing to motion. Friction in engine pistons 
wastes fuel, as does tire pressure and extra luggage. All told, as little as 14 
percent of passenger car fuel is converted into useful energy. Clearly, this gas 
guzzling engineering throwback is rife for disruption from electric cars and 
advances in battery technology—time will tell. 


In many ways software development is as wasteful as the internal combus- 
tion engine. Friction between development and operation causes delays. 
Manually assembling multiple components and configurations within our own 
software factories leads to lost time. Carrying excess inventory in the form 
of unneeded infrastructure capacity adds extra cost. Add to this constraints 
preventing access to critical systems and data during testing, and defects can 
accumulate across the software lifecycle. 


Lean and Value Creation 


The traditional view of IT value has been internally-shaped. For decades, sys- 
tems and applications have been designed, built, tested, and released to cus- 
tomers, citizens, and end users where they hopefully influenced behaviors. All 
this has changed. With the advent of cloud, mobility, and social computing, 
consumers rather than producers call the shots. This means businesses find 
themselves in the position of having to respond to the behaviors and desires 
of their customers. 


For IT, this redefinition of business value means teams must focus on two 
essential strategies. First, they must continuously reexamine the software ser- 
vices they deliver from the perspective of the customer, and second, they 
must constantly strive to minimize any interference or waste across the entire 
software factory. This includes everything that impedes the flow of value to 
customers and incurs more cost. 


This notion of value being “pulled” by customers and waste elimination is not 
new. Lean pioneers and practitioners such as Toyota, Motorola, and Xerox 
redefined manufacturing by applying these principles; understanding that many 
forms of waste exist across production processes. And, because they add no 
value to customers, must be clinically removed. 
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But can Lean principles be applied in IT, where, unlike traditional manufactur- 
ing, waste isn't visible across a factory floor through telltale signs like excess 
physical inventory or idle machinery? Often due to its intangible nature, waste 
in IT can be hard to identify yet alone eliminate. 


Interestingly, the delivery of software bares many similarities to a manufac- 
turing process. In IT, we have the means to respond to value triggers from 
our customers by quickly designing, developing, and releasing software ser- 
vices. And, since the software delivery lifecycle represents a manufacturing 
production line within a software factory, we have the guiding context upon 
which identify all elements of waste that add no value to the business and its 
customers. 


Eight Elements of Waste 


As illustrated in Table 3-1, there are eight elements of waste or “Muda” (using 
Lean terminology) that severely impact the IT group's ability to increase the 
value of software services. 


Table 3-1. Eight Elements of Waste (D.O.W.N.T.I.M.E) 


Type of Waste Examples Business Outcomes 


Defects Badly designed and poor quality code Lost customers and revenue; 


Non-functional performance issues Negative brand impact 


Overproduction Delivering features customers don’t Delays, cost over-runs, and 
need or want budget problems 


Procuring extra capacity due 
to unanticipated performance 
requirements 


Waiting Excessive release backlogs and Slow time-to-market and 
bottlenecks value; lost opportunities 


Infrastructure and data not available 
for testing 


Change reviews; security and 
compliance audits 


Non-value added Lengthy problem resolution and Morale issues; high-staff 
processing fire-fighting turnover 

Team-based activities prioritized over 

program-level objectives 
Transportation Frequent release rollbacks Application launch delays; 


Development/QA handoffs increased cycle times 


(continued) 
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Table 3-1. (continued) 


Type of Waste Examples Business Outcomes 


Inventory (excess) Underutilized resources Increased capital and 


Partially completed work and operational costs 


excessive work in progress 


Motion Developers constantly switching Lost productivity; talent 
Relearning and rework erosion 

Employee Closed retrospectives and stand-up Missed opportunities to drive 

knowledge (unused) meetings improvements 


No feedback established from 
service management (e.g., call center/ 
service desk) 


Examining this table it should be noted that there are close relationships 
and linkages between elements. For example, undetected code defects 
resulting in performance problems may result in an organization purchas- 
ing additional hardware capacity, which leads to excess inventory, which 
increases the support burden. In situations like this, waste begets waste 
and technical debt accumulates to such an extent it becomes difficult to 
pay off. The result is that essential development is tied up on maintenance 
and support activities. 


Originally coined by Ward Cunningham in the Agile Manifesto, technical debt 
has tended to be reviewed from a development perspective’. After all, if soft- 
ware defects can be identified and eradicated during early stages of develop- 
ment, then production related problems (which could be significantly costlier 
to fix) can be avoided. 


But technical debt can also be created in IT operations. For example, failing 
to document or visualize business services (and supporting applications an 
infrastructure) means teams could take longer triaging problems. Here the 
waste is non-value added processing, which again (because of linkages) results 
in more waste. In this case, increased transportation because a release has to 
be rolled back. 


Using the eight elements of waste list, DevOps practitioners can begin a pro- 
cess of identifying waste elements across the software lifecycle. It's important 
to understand that "toxicity" levels will vary, so mechanisms must be devel- 
oped to continuously reveal new situations and conditions that can potentially 
introduce more waste. 


‘The Agile Manifesto: http: //www.agilemanifesto.org/ 
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It's also critical not to restrict the exercise to new development. These may 
become the debt burden of the future, but could only represent a small part 
of the portfolio. Legacy infrastructure and production applications should also 
be included because, even though they change less frequency, they often incur 
significant management costs and overheads. 


Finally, debt and the associated waste should be reviewed as a continuum, 
with special attention paid to integrations between new customer facing apps 
and essential back-end business processes. For most enterprises, multi-chan- 
nel engagement creates tremendous opportunity for value creation, but will 
introduce more waste if they're not integrated and coordinated with existing 
back-end systems, applications, and call-center services. 


Waste Removal Strategies 


Today's mobile and API-centric forms of service delivery mean that custom- 
ers assess value based on extremely high levels of functional and operational 
quality. They also expect businesses to deliver additional value in the form of 
continuous change. With customer experience so important, it's critical to 
begin waste identification from the perspective of the customer; monitoring 
and analyzing the usage and behaviors of applications and determining what 
elements impact the total experience.This is especially important for mobile 
applications, since factors beyond the control of IT departments (e.g., car- 
rier network latency and cloud service performance) can quickly erode value, 
however good the functional quality. 


Some immediate practices cross-functional teams can apply to help identify 
and eliminate waste, include the following. 


Prevent Defects by Removing Constraints 


When development and testing is constrained due to lack of access to depen- 
dencies (e.g., middleware, web services, and test data), defects can quickly 
work their way into the code base. This is illustrated in a ServiceVirtualization 
survey, which identified that on average, participants require access to 52 
dependent elements for development or testing, yet have unrestricted access 
to only 23 of these.° 


To circumvent these issues, many development teams often attempt work- 
arounds by hand coding (mocks and stubs), but this doesn't provide for 
realistic application behavior, causing test validation errors and the late 


*VOKE Market Snapshot™ Report: Service Virtualization; https: //www.ca.com/au/ 
register/forms/collateral/voke-market-snapshot-report-service-virtual 
ization.aspx 
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discovery of defects. Discussed further in Chapter 5, a more scalable 
approach is to incorporate Service Virtualization into parallel develop- 
ment and test activities. 


Focus on Value to Prevent Overproduction 


New application features don't necessarily mean more customer conversa- 
tions and increased revenue. Unnecessary features can result in additional 
maintenance overheads and cost. There are many methods DevOps practi- 
tioners can use to reduce this form of waste, including: 


e Incorporating application experience analytics into moni- 
toring strategies to identify mobile app functions and fea- 
tures that are not used 


e Split or A/B testing and funnel or cohort analysis 


e Refactoring code elements to reduce complexity, remem- 
bering that the cheapest and most reliable components 
are those that don't exist! 


Smoothing Flow to Reduce Wait Times 


Like waste element £l, this waste can eventuate due to delays waiting on 
dependencies during development and testing. In the YOKE report mentioned 
above, 8| percent of participants identified development delays of waiting 
for a dependency in order to develop software, reproduce, or fix a defect. 
Additionally, 84 percent of participants identified QA delays of waiting for a 
dependency in order to begin testing, start a new test cycle, test a required 
platform, or to verify a defect. 


Caution Never underestimate the wait times associated with accessing test data, since a 
massive 20 percent of the average software delivery lifecycle is wasted waiting for data, locating 
it, or creating it manually when none exists. 


Excessive wait times may also be due to problems managing highly complex 
release and deployment processes. However good the code, its ultimate value 
will be determined by how quickly it can be deployed into production. Manual 
processes and fragile scripting not only compromise these goals, but also 
increase the potential for defect code being released. These issues can be 
addressed by: 


e Ensuring all key stakeholders possessing the knowledge 
to move a service swiftly across lifecycle are involved 
early and often 
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e Using smaller batch sizes so that value is delivered to 
customers at regular intervals 


e Developing and automating reusable and repeatable pro- 
cesses to simplify and streamline application releases 


Limit Non-Value Added Processing Through Data-Driven 
Insights 


Fixing application problems provides limited value to customers. Rather than 
wait for problems to occur in a production, IT operations should be involved 
much earlier in the development lifecycle. 


Using tools to share information is especially valuable. By leveraging applica- 
tion performance change impact analysis during a build process, for example, 
developers can quickly determine any adverse performance conditions their 
code is introducing. 


Reduce Transportation Cost by Automating Deployments 


When work is manually handed off from one team to another (e.g., developer 
to test/QA, QA to operations), critical knowledge can be lost. This could 
lead to additional delays or the highest transportation cost of all—release 
rollbacks. 


There are many strategies to address these issues, including: 


e Reducing the number of handoffs by automating standard 
tasks and activities 


e Ensure release automation tools provide an extensive 
set of action packs and plug-ins so as to fully deploy at 
an application level, while also integrating key supporting 
processes (e.g., configuration management) 


e Build more knowledge as releases progress (e.g. 
establishing application performance management in 
pre-production) 


Eliminate Excess Inventory Across the Software Factory 


Minimizing inventory is the hallmark of Lean thinking. As in traditional manu- 
facturing, there are many waste indicators in IT's own software factory. In 
development, partially completed work can become obsolete before it finds 
its way into production and should be exposed to ensure it doesn't degrade 
or corrupt the code base. In operations, excess on-premise server infrastruc- 
ture acquired as a fail-safe to address unanticipated performance problems 
could be avoided by establishing monitoring in pre-production. 
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Note Costs can accumulate substantially when agile teams acquire specialist tools. Work 
collaboratively to assess whether the additional cost (training or support) offsets the value delivered 
to one team. 


Prevent Unnecessary Motion with Parallel Development 


While transportation waste is associated with the unnecessary movement 
of software, motion waste involves the unnecessary movement of people. A 
good example is task switching, where an API developer might shift focus to 
a new project rather than wait for testing dependencies to become available. 


Apart from adding more waste (e.g., delays), task switching can introduce 
many more problems, especially related to the productivity of developers due 
to constant interruptions. 


Some simple strategies to reduce this waste, especially task switching, include: 


e Try to ensure teams have all of the knowledge, tools, and 
data needed to complete their assigned work 


e Simulate and virtualize all dependencies so that develop- 
ment teams can code and test in parallel 


e Since as much as 50 percent of testing is wasted by teams 
trying to locate test data or create it manually, consider 
supplementing constraint-removal strategies with test 
data management (see Chapter 5) 


e Aim to eliminate unimportant work, meetings, and inter- 
ruptions. If it isn't delivering value, ask why your team is 
doing it! 


Incorporate Employee Knowledge Using Feedback Loops 


While the feedback of production information is important to drive software 
improvements and improve supportability, it isn't the only place where knowl- 
edge can be transferred. 


Service desks and call-center processes should also include mechanisms 
to deliver (to development) important information gained from custom- 
ers on their usage and response to new application features and functions. 
Knowledge transfer should also be bi-directional. For example, application 
experience analytics could (when integrated with incident management pro- 
cesses or even social media) become an early warning mechanism to trigger 
coordinated responses in the event of mobile app usage problems. 
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DevOps Metrics 


With any IT-driven methodology or program, measuring the effectiveness 
in a business context is critical. But since DevOps isn't a formal framework, 
organizations have little guidance in determining what metrics should be 


used. 


This can be problematic and lead to a number of suboptimal practices: 


Efficiency status-quo—The IT team falls back to metrics 
traditionally used to demonstrate technical proficiency in 
meeting stability and resilience goals. Although these are 
not necessarily wrong, DevOps metrics should also dem- 
onstrate how new processes and automated technolo- 
gies are impacting the business—for example by speeding 
time-to-market and reducing lead times. 


Outputs over outcomes—Organizations gravitate to 
metrics that are commonly used in assessing team- 
level productivity. These can include output-based 
metrics like number of features delivered or servers 
provisioned. Metrics in this class can be counterpro- 
ductive unless balanced with outcome-centric indica- 
tors that show results achieved against desired quality 
levels. 


Low-hanging fruit—Organizations select metrics that are 
easily obtained but not necessarily useful. Since DevOps 
success is predicated on cultural change, businesses must 
also measure what's harder to determine but potentially 
more valuable—namely, how the adoption of DevOps 
behaviors and values at an organizational level is impact- 
ing the business. 


Anti-Pattern Metrics 


Before embarking on a metrics refresh, organizations should consider all exist- 
ing measures and their applicability in a DevOps context. Particular atten- 
tion should be given to carefully review those metrics and incentives that are 


counter to DevOps principles, as illustrated in Table 3-2. 
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Table 3-2. Problematic Metric Classes 


Metric Class Examples Adverse Effects 
Vanity Metrics ^ Lines of code May be counterproductive since they reward the 
produced wrong types of behavior—especially if incentives 


Function points are linked to the metric. 


created Producing more code and features without 
validation can inhibit other valuable activities such 
as refactoring and design simplification. 


Intra- Team Agile team Beware of metrics that pit-teams against 
Metrics leaderboards each other and use vanity metrics as scoring 
mechanisms. 


Deployments/ 

changes prevented Strike a balance with metrics and rewards that 
influence positive inter-team behaviors—such as 
code sharing, peer reviews, and mentoring. 


Pay particular attention to metrics that promote 
an anti-DevOps culture, such as rating operational 
effectiveness on the ability to prevent releases and 
deployments. 


Traditional Mean-time- With faster delivery of services, some failure is to 
Metrics between-failure be expected. 
(MTBF) Always consider that improving responsiveness 


FTEs: Servers can be more important (and less costly) than 
trying to prevent failures. 


Suitability Checklist 


When reviewing and developing DevOps metrics, it's also important to con- 
sider each against a general suitability checklist: 


e  Obtainable—Culture and behavioral improvements are 
important to measure, but metrics may be difficult to 
obtain or quantify. Seek out other related data points to 
help expose —e.g., staff retention rates/transfers as an 
indicator of employee morale. 


e  Reviewable—Every metric must stand up to rigorous 
scrutiny in a business context. Carefully review metrics 
that can be easily collected, but add no tangible value— 
e.g., lines of code produced per developer. 


e  Incorruptible—Determine whether each metric can be 
influenced by team and employee bias. Seek out any 
associated incentives that can work against a collabora- 
tive DevOps culture—e.g., existing SLA bonuses inhibit- 
ing change. 


Wherever possible, metrics should also be shareable and have relevance 
across the software lifecycle to both development and operations. For exam- 
ple, generating security scores at a cross-functional team and divisional level 
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Actionable—Any metric must support improved decision 
making. Exposing A/B testing results can for example be 
a valuable way to quickly determine the effectiveness of 
new functionality. 


can be used to inform teams about the risks of their actions. 


Metrics that Matter 


Having determined what not to measure, the next stage is to develop a can- 
didate list of metrics supporting the DevOps program. One common mistake 
is to measure too many elements, falling back to what's easily collectable. 
Additionally, metrics applicable to DevOps may be new to organizations (e.g., 
the speed of deployment, rate of change, and customer responsiveness), so 
it’s important to think broadly how changes to work practices, process and 


technology can support these goals. 


When developing metrics, it’s important to maintain balance. Defaulting to 
metrics skewed toward one particular area (e.g., operational or development 
efficiencies) can have a negative effect in terms of behavioral improvement 


Figure 3-1 illustrates four dimensions and sample metrics that can be used to 


People—Staff related metrics can be the most difficult to 
collect but are still powerful change indicators. Strong 
consideration should be given to internal metrics like 
staff retention rates and training, together with mentor- 
ing and knowledge building (e.g., open source contribu- 
tions and wiki development). 


Process—lt's important to consider how existing prac- 
tices will help or hinder new targets being achieved, pay- 
ing special attention to existing bottlenecks (e.g., security 
audits only conducted after testing will impact deploy- 
ment rates). 


Technology—Good metrics are those that help teams 
drive improvements, even after failures (e.g., what is the 
percentage of failed releases and what percentage of 
these were due to code defects, manual processing, con- 
figuration errors, etc.). 


measure the effectiveness of a DevOps initiative. 
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Customer & Business Value 


Lead times 
Revenue per user story 


NPS (Net promoter scores) 
Customer conversion (by App / function) 


Efficiency & Effectiveness Quality & Velocity 


cle times 
FTE to Customer ratios Cost per transaction / app MI = 


Rollback rates Deployment frequency 


Change/ release cost burden 
Operational support costs 


Culture, Collaboration and Sharing 
Staff retention Morale/ job satisfaction Wiki / open source contributions Mentoring 


Figure 3-1. DevOps metrics dimensions 


Culture, Collaboration, and Sharing 


Metrics in this category are especially valuable because they provide an ongo- 
ing indicator of acceptance/resistance to DevOps. Some metrics in this dimen- 
sion will be easier to collect (e.g., staff retention rates/turnover) than others 
(e.g., employee morale). It's important therefore to look at measures across 
other dimensions to understand how they impact this area. For example, 
are mean-time-to-recover (MTTR) improvements positively impacting staff 
morale, absenteeism rates, and responsiveness to change? Consideration may 
also be given to automated surveys and employee feedback, as long as these 
are fully transparent and actionable. 


Efficiency and Effectiveness 


Metrics here normally focus on elements of development capacity and opera- 
tional capabilities. While traditional metrics such as server to sysadmin ratios 
have been used, many organizations are now adopting more customer-centric 
ratios like full-time-equivalent (FTE) to customers. 


Examining full costs on a transactional or application basis is another good 
candidate metric, as it's focused on improving data center efficiencies (e.g., 
energy and cooling). Other metrics such as cost of release are also good 
since these can expose inefficiencies associated with acquiring, prepar- 
ing, and maintaining physical infrastructure for development, testing, and 
production. 
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Quality and Velocity 


This dimension looks to measure data points with respect to service deliv- 
ery. For organizations starting on a DevOps initiative, many indicators (e.g., 
percentage of deployments rolled-back due to code defects/outages/negative 
user reactions) could initially be high. This may be a result of the extra time 
needed to adopt new processes, combined with remediating existing techni- 
cal debt and waste elements these metrics expose. However, with DevOps' 
focus on establishing quality right from the start of development, this should 
reduce over time. 


When paired, these metrics also provide additional insights. For example, if 
the rate of rollbacks still increases during periods of low change volume it could 
be indicative of serious problems, e.g., errors due to manual/scripted release 
processes, task switching, and excessive handoffs. 


Other useful metrics in this dimension include: 


e Cycle time—Measures the length of time it takes to com- 
plete a stage or series of stages in a release operation. This 
can be extremely valuable in exposing any bottlenecks. 


e MTTR—This can be broken down into detection, diag- 
nosis, and recover phases. MTTR is a great indicator of 
how effective teams are in handling changes. For complex 
deployments, there will be spikes, but this metric should 
be trending down as DevOps becomes established. 


Customer and Business Value 


This category of metrics are externally focused and help measure how 
DevOps supports business goals—like increased customer loyalty and faster 
time-to-market. The manufacturing concept of lead time provides DevOps 
practitioners with an analogous metric (time taken from when code starts 
development to successful production deployment) and determines how well 
DevOps is at meeting the need for rapid delivery of high-quality software ser- 
vices. This metric is especially important to scrutinize because long lead times 
could be indicative of code defects or testing constraints. 


Another interesting candidate is Net Promoter Score (NPS), which is a sim- 
ple management method to measure customer loyalty. While this metric has 
traditionally been used in other areas of the business (e.g., marketing), its 
inclusion is valid since the loyalty of customers is increasingly determined by 
how quickly high-quality software services and updates can be delivered to 
via web sites and/or mobile apps. 


46 


Chapter 3 | DevOps Foundations 


Additional Methods and Techniques 


With metrics developed across each of the four dimensions discussed previ- 
ously, teams can begin a process of determining the relationships between 
them. This is important so teams gain insight into what processes enhance- 
ments and tools are needed to meet targets or address capability gaps. 


One simple and effective approach, as illustrated in Figure 3-2, is business 
impact mapping. This involves determining which DevOps processes will be 
needed to support a business or customer experience goal, together with the 
underpinning metrics, targets, and initiatives/tools across multiple dimensions 
that support this outcome. 


Business Metric/Goal DevOps Focus Metrics - Targets Supporting 
Initiatives 


Cycle Times — 
reduce by n% Release Automation 
CONTINUOUS 
DELIVERY 


Software Defect Rates — Test-driven 
Net Promoter reduce by n% development 


Score (NPS) 


(achieve 8+) 


Response times — Transaction tracing 
n% faster 
AGILE 
OPERATIONS Application 


Restoration times — Performance 
speed MTTR by n% Management 


Figure 3-2. Metrics, targets, and initiatives linked to business outcomes 


Figure 3-2 illustrates that an organization is seeking to achieve a Net Promoter 
Score of 8*. To support this goal, IT needs to deliver software releases and 
new functionality faster, together with ensure a high-quality customer experi- 
ence. Metrics and targets have therefore been set within the quality and veloc- 
ity dimension, together with targets and supporting process/tool initiatives. 


As DevOps metrics programs develop, practitioners should also: 


e Have regular and ongoing target reviews to ensure that 
goals are not completely unrealistic or that existing pro- 
cesses and tools are not delivering improvements. 


e Consider removing persistent "green light" metrics when 
targets have been consistently achieved. 


e Avoid having every metric focused on velocity without 
paying attention to customer satisfaction and loyalty. 
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e Strive to prevent vanity metrics and operational or team- 
centric bias creeping back into the program and distort- 
ing the true performance picture. 


e Beware of ranking teams based on targets—the best way 
to compare teams is to measure things like customer loy- 
alty (as described) and how successful teams are in meet- 
ing their commitments. 


e Give strong consideration to metrics, targets, and initia- 
tives that foster peer review and openness. 


e Carefully build incentives and reward programs that rein- 
force the value of a strong collaborative culture. 


e Involve business counterparts right from the start to 
ensure customer and business data is held to the same 
standard as operational/efficiency metrics. 


e Match tools to the DevOps program, especially those 
that can monitor and respond to real-time conditions 
(such as transaction times, response times, and mobile 
app crashes), but can also proactively detect and prevent 
adverse conditions (such as code defects and release bot- 
tlenecks) that impact performance. 


Summary 


At its heart, DevOps is about building a generative organizational culture 
where business improvement is placed above everything else. But as this 
chapter has illustrated that won't always be straightforward, especially in 
organizations beset by divisional friction and lack of direction. By leveraging 
this chapter's guidance, especially with regard to building high-trust teams, an 
outcome-based metrics program and Lean thinking, organizations have a solid 
foundation upon which to guide their DevOps programs. 


In Chapters 4-7, we'll look closely at the automated tooling needed to sup- 
port this goal and how businesses can refit and re-engineer their own soft- 
ware factories to manufacture high-quality software innovations, at speed. In 
these chapters, we'll examine critical tooling strategies across the software 
lifecycle continuum— Build, Test, Deploy, and Manage. 


LAE 


Essential 
DevOps Tooling 


CHAPTER 
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Build 


APIs for the Connected Business 


In the digital economy, building application programming interfaces (APls) is 
essential for executing ideas quickly and seizing new business opportunities. 
APIs are the building blocks of digital transformation, enabling organizations 
to deliver exceptional customer experiences, create new revenue streams and 
connect employees, partners, apps, and devices to data—anytime, anywhere. 
APIs are not necessarily a new technology, but in today's digital world, they 
have risen in prominence and become important to every facet of the enter- 
prise. This in turn has increased the demand for effective API management. 


But what does an effective API management look like? A good place to start 
answering this question is by examining a case study. 


Case Study: IceMobile 


IceMobile helps leading food retailers across the globe boost footfall and bas- 
ket size. Ralph Cohen founded the then creative agency in 2002 after recog- 
nizing that mobile devices offered a great potential for increasing customer 
engagement, loyalty and, ultimately, spend. ' 


In 2012, IceMobile took two steps that transformed its business into the global 
mobile loyalty program provider it is today. First, it merged with BrandLoyalty, 
extending its reach beyond the Netherlands. Secondly, it built the Bright Loyalty 


'Full story:http: //www.ca.com/content/dam/ca/us/files/case-studies/icemobile- 
cuts-rollout-times-with-ca-api-gateway.PDF 
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Platform, which helps food retailers engage with shoppers on a personal level, 
driving turnover, increasing basket size and bringing in new customers. 


At the solution's front-end, the Bright Stamps product makes loyalty cam- 
paigns digital.Shoppers can easily collect stamps and redeem using their mobile 
phone, resulting in increasing spend and more people joining the campaigns. 
At its back-end, the Bright Loyalty Platform analyses shopper data from retail- 
ers' back office point of sales (POS) and customer relationship management 
(CRM) systems to ensure the shopping experience resonates with customers. 


Located in offices across 14 countries, IceMobile's 120 developers, design- 
ers, and engagement experts offer retailers a range of loyalty services. These 
include concept and strategy, user experience design and research, solution 
design, implementation, project management, and data analysis. 


When IceMobile talked to retailers regarding its product Bright Stamps, despite 
commercial departments being enthusiastic about adding the Bright Stamps 
solution to their loyalty programs, retailers' legacy IT systems were often a 
hurdle to overcome. "Retailers were concerned their IT departments, which 
were often starved for time and budget, would not have sufficient resources 
available for integrating IceMobile with their various legacy systems,” reveals 
Jeroen Pietryga, IceMobile's Chief Executive Officer. 


The company needed to find a safe and manageable way of accelerating and sim- 
plifying the integration between its Bright Stamps solution and retailers' back 
office systems. “We wanted to take some of the IT headaches and risk away 
from retailers’ IT departments," adds Pietryga. “We wanted to make imple- 
mentation faster, cheaper, and a better experience?" Application Programming 
Interfaces (APIs) gateway provided IceMobile with the perfect solution. An 
API gateway solution would only require limited changes to the food retailers’ 
back office systems while reducing the amount of time IceMobile engineers 
needed to spend on implementing the integrations. 


By simplifying integration, IceMobile has cut implementation times from 14 to 
8 weeks while minimizing impact on retailers' busy IT departments. 


The solution also ensures IceMobile meets security standards for inter-oper- 
ating with retailers’ IT systems; ensuring the digital stamps IceMobile collects 
and stores for customers are safe. 


The API gateway also enables IceMobile to: 
e Minimize impact on its own and retailers’ IT departments. 


e Integrate with almost any technology used by a retailer, 
safeguarding sales. To date, the solution has simplified 
connections with multiple back office systems for various 
food retailers. These back office systems include POS sys- 
tems, such as Wincor Nixdorf, NCR, and IBM, and CRM 
systems, such as Siebel, eFuture, and SAP. 
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e Standardize the way it delivers its services as it expands 
across the globe. 


From Little API Acorns Big Things Grow 


Like IceMobile, many companies have wisely invested in an API strategy and 
are reaping the benefits. Of course, this includes some pioneers, like Netflix 
and the streaming of content to over 200 devices types, thanks to APIs, or 
Twitter, who receives a staggering 13 billion API calls to its platform each day. 
But it's more about big business than big numbers. APIs enable Expedia to 
conduct $2 billion of business through its partner affiliate network, and 60 
percent of eBay's listings are made through its API. 


While software-driven companies have much to benefit others in the 
"physical product" world have realized the API carpe diem moment. Withings 
have transformed the humble bathroom scale into a hub for a fitness eco- 
system, while Nest Labs' thermostat is a marvel in intelligent home energy 
management. 


So irrespective of whether business sell sedans, soda, or sneakers, executing a 
successful API strategy can deliver numerous advantages. 


e Customer satisfaction—No matter how fast the new fea- 
tures are released, customers still expect more. By expos- 
ing applications and data to external partners, developers, 
and customers, APIs can significantly ease the feature and 
release pressure on IT teams. 


e Scale and reach—With API enabled networks, business can 
work with partners to build new sales channels without 
relying on existing functions to generate and support them. 


e Business efficiency—Customers look to the provider for 
specific customizations. If each of these requests had to be 
addressed, development could be distracted from deliv- 
ering more strategic features across the customer base. 
By exposing functionality through APIs, implementation 
teams and partners could provide this as a differentiated 
service, which is another revenue generating opportunity. 


APIs provide the essential glue that binds together internal business depart- 
ments, allowing them to operate as a seamless whole. They enable resources 
and information to be decoupled from functional silos and made available for 
greater business benefit. So if sales needs access to marketing data or market- 
ing needs customer analytics, the secret sauce will be an API. 


This is extremely beneficial from a DevOps perspective. Since APIs help unify 
people, process, and technology across the software lifecycle, every group 
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becomes a partner of the other. If development needs access to application 
performance information from production monitoring tools, it's an API that'll 
service the request. If a production-based API needs to be tested against 
changes to a back-end system, it'll be another API (together with service visu- 


alization software) that helps removes the constraints. 


API Management: Stakeholders and 
Requirements 


Because different stakeholders perceive APls in different ways, they will 


require specific management capabilities in context of their role. 


Business executives-APls and management must be a 
strategic enabler for launching new innovative products, 
forging new business partnerships and improving the cus- 
tomer experience. These stakeholders will be especially 
interested in planning and tracking API revenue and ROI. 


API owners/product managers—This group will have respon- 
sibility for creating and owning an API strategy and road- 
map. As such, they be subject matter experts and oversee 
how APIs are released and managed. They'll also review 
API roadmaps with key stakeholders to determine API 
priorities, including new enhancements. 


Enterprise architects—Being responsible for translat- 
ing digital initiatives into an optimum technology infra- 
structure, they'll regard APls as the connective tissue. 
API management will provide tools to help them model, 
design, shape, and optimize critical integrations across 
both legacy and modern infrastructure. 


Application developers—Developers build front-end appli- 
cations, while discovering, acquiring, and consuming APIs 
as their gateway to enterprise data and capabilities. For 
this group, API management represents stable, secure, 
and scalable access to both cloud and on-premise sys- 
tems, with tools that help them leverage APIs quickly by 
removing development and testing constraints. 


Security managers—Because APIs “open the enterprise,” 
they create new security challenges and opportuni- 
ties. For this group, API management is about providing 
advanced threat protection, data encryption, and authen- 
tication, all without compromising the goal of increasing 
connectivity and convenience. 
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e API support group—Regardless of which team supports 
APIs (development or operations), advanced monitor- 
ing will be needed to ensure consumers have optimum 
access to the API and related services (e.g., back-end sys- 
tems and cloud infrastructure). 


Note Because of the importance of APIs, many companies organize teams around their 
development and ongoing support. Rather than engage multiple stakeholders on an as-needed 
basis, dedicated cross-functional API teams are established. This can be a very effective DevOps 
practice since it removes functional silos and aligns teams around desired business outcomes— 
versus managing APIs in technology silos. 


APIs Are Products 


APIs are the building blocks of great products. With APIs, modern applications 
can be developed faster and integrating existing systems becomes easier. And 
because APIs can help facilitate new partnerships and business opportunities, 
they can in fact be considered actual products in their own right. 


As with any software product, APIs will go through a lifecycle where they 
are designed, created, secured, managed, and optimized. With revenue and 
reputation increasingly riding on APls, this will need to be conducted on an 
enterprise scale. API management provides organizations the ability to achieve 
this, as long as capabilities embody the principles of DevOps and continuous 
delivery. While many individual tools address the discreet needs of the indi- 
vidual stakeholder groups, the goal of API management should place overall 
business improvement above everything. 


Managing the API Lifecycle 


API management embodies the principles of DevOps by managing the API 
lifecycle; providing the platform, visibility, and tools that architects, developers, 
security professionals, and administrators need to drive continuous delivery. 
Without effective API management, organizations will increasingly struggle to 
deploy, control, measure, and optimize the growing volume and variety of APls 
needed to support digital transformation initiatives. 


Since API management inherently addresses the DevOps lifecycle for APIs, it 
should address all elements (see Figure 4-1) and empower every stakeholder 
described in this chapter. 


It should be noted from Figure 4-| that the definition, design, deployment, pro- 
motion, protection, and optimization of APIs is not limited to just the develop- 
ment team. [T operations and business stakeholders also need to be involved 
to ensure APls are aligned to the overall program initiatives. 
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API Consumption API Provision 


Initiate 
Iterate 


Figure 4-1. API management lifecycle 


Ongoing measurement and monitoring is often overlooked in the API lifecycle. 
For a DevOps-driven API strategy to succeed, organizations need to be able 
to evaluate API utilization and performance as applications evolve; answering 
questions, such as who is consuming APls and how often are they doing so? 
Which APIs are meeting business goals? Where are the latency issues? Will 


the API scale to support demand spikes? 


Analytics play a critical role in answering these questions. In a DevOps con- 
text, they are also especially valuable in serving essential information and feed- 
back to cross-functional team members in order to drive API improvements. 


This includes: 


Automation and integration are essential for capturing such insights with mini- 
mum effort. Effective API management is founded on intelligent and integrated 
tools that simplify different stages of the lifecycle for different stakeholders. 


Performance analytics—Providing developers with real- 
time insight into operational performance, such as trans- 
action speed, availability, and API latency. This helps ensure 
apps are meeting their users’ performance expectations, 
particularly during peak usage periods. 


Business analytics—Help organizations monetize APIs by 
tracking usage and consumption by external partners 
and developers. This means they can better respond to 
market demand and prioritize future development invest- 
ments in APIs. 


App experience analytics—Report on metrics, such as app 
usage and revenue generation over the lifecycle of the 
app; providing complete return on investment visibility 
both in the short and long-term. 
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For developers, API management solutions need to provide a simple point and 
click interface that can automatically generate scalable enterprise-grade APIs 
from data sources as diverse as RDBMS, NoSQL, existing APIs, and JSON. 


This is particularly important as legacy monolithic systems are supplemented 
with dozens of microservices, each fronted by an API. While microservices 
decrease the size of each deployment, they increase the number of com- 
ponents deployed. API management tools are vital for helping developers 
navigate the microservices maze: first by auto-generating APIs and then by 
managing and providing simple yet secure access. 


Access and discovery can also be simplified by leveraging an API gateway, 
which frees up time for developers to focus their efforts on delivering what 
the customers want—useful and engaging apps that work on multiple devices. 


Essential API Management Plays 


While API management provides capabilities to many stakeholders, the stron- 
ger methods will be those that unify teams toward continuously improving 
business outcomes from the API strategy as a whole. By reviewing their stra- 
tegic API objectives (e.g., accelerate mobile development), organizations can 
assess API management capabilities using a number of tactical initiatives, or 


"plays". 


Create and Integrate APIs 


In the past, organizations have relied on APIs from SQL calls and hand-coded 
business logic—often developed on an ad hoc basis. However, new digital 
imperatives now dictate adopting scalable and sustainable approaches to cre- 
ating API-based apps faster and unlock the value from legacy applications and 
disparate data stores. And with DevOps processes needing to support the 
development of microservices that can be changed faster, this pressure will 
only intensify. 


To address this, API owners, enterprise architects, and business executives 
should consider adopting methods and API management solutions that expand 
the scope of the API lifecycle management beyond gateway enforcement con- 
trol toward the creation of APIs closer to business information. 


The value proposition here is a significant reduction in the time and cost 
associated with manual-based API creation. Combine this with immediate 
API performance monitoring and security (discussed next) means quality and 
compliance becomes established as APls are created. 
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As illustrated in Figure 4-2, the API management needed to facilitate rapid API 
creation and integration should include: 


e Connect SOA, ESB, and legacy applications—The ability to 
streamline integrations across disparate systems, middle- 
ware, and databases by providing protocol adaptation, 
mediation, and transformation. 


e Aggregate data from multiple sources—Assist engineers 
integrate SQL and NoSQL databases with fine-grained 
data access controls and the flexibility to deploy and scale 
to current and future architectures. 


e Connect cloud services—Enable the creation of performant, 
yet cost-effective API-centric digital platforms that integrate 
and orchestrate on-premise systems and cloud solutions. 


e API creation from reactive logic—Instantly generate enter- 
prise-grade REST APIs from multiple data sources, with 
logic-based processing to apply business rules to API calls 
at runtime. 
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Figure 4-2. API Management for API creation and integration 


Note Using reactive logic models to assist in the creation of APIs can be significantly faster 
than hand-coding. These techniques are valuable to DevOps practitioners since concise application 
logic is easier and less costly to support. 
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Secure the Open Enterprise 


As more information assets and engagement channels become digitized, secu- 
rity is a major concern across the API lifecycle—from the business owner and 
the developer to the user. 


For developers, API management represents stable, secure, and scalable 
access to back-end systems and information stores, as well as a source 
of tools and utilities to help them obtain and leverage APls more effi- 
ciently. API management allows them to simply consume existing secure 
encrypted APls, for example, without needing to scribe a unique security 
module for each. 


In a DevOps context, information security professionals will work with devel- 
opers to identify and neutralize critical threats, enable robust and workable 
policies, offer consistent and repeatable security for mobile apps, and provide 
the capabilities needed to deliver features such as single sign-on and privileged 
user access. The goal shouldn't be to wait for security issues later, but rather 
to help teams establish strong security as they design, create, and test APIs. 
This may include 


e Protection against threats and vulnerabilities—AP| manage- 
ment will provide threat detection and neutralization for 
key Open Web Application Security (OWASP) vulner- 
abilities such as SQL injections, cross-site scripting, and 
denial-of-service attacks. 


e Controlled access with SSO and identity management— 
Securing apps and their connections, while maintaining or 
enhancing the all-important user experience. 


e Providing end-to-end security for apps, mobile and loT—API 
management will protect the digital value chain from 
front-end app to back-end systems. It should extend con- 
trolled access to all touch points —from web apps to loT, 
while supporting convenient features such as social login 
and risk-based authentication. 


Unlock the Business Value of Data 


Competitive pressure, rising customer expectations, and the increasing pace 
of change mean that applications—especially for mobile and the loT—must 
be delivered faster and more efficiently. Developers look for API management 
to help them discover, acquire, and consume APIs quickly, while also provid- 
ing tools that speed up or eliminate the “dirty work" of repeatedly building 
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core functionality to handle data and security. Therefore, comprehensive API 
management should support: 


e Simplified and controlled access to data—APl management 
will provide a controlled way to access data that shields 
developers from unnecessary complexity. It should aggre- 
gate and orchestrate data, while ensuring compliance 
through authorization, shaping and policy management. 


Tip To protect customer privacy and ensure compliance during API testing, consider integrating 
with management solutions that provide synthetic test data. 


e Support a wider partner/public developer ecosystem—Solutions 
will empower internal and external developers by stream- 
lining API consumption lifecycle tasks such as discovery, 
acquisition, design, and collaboration. 


e Reduce mobile app delivery time—Developers will need 
access to reusable services in the form of SDKs and APIs 
that provide security, messaging, and offline storage. 


Accelerate Mobile and loT Development 


Digital transformation initiatives that leverage APIs create new business 
opportunities and channels. Businesses look to API management as a central 
launch point for their digital strategies, with a range of capabilities that will 
support their efforts to build a robust digital ecosystem by expanding partner- 
ships, nurturing developer communities, monetizing data, and leveraging digital 
connections to improve operations and efficiency. 


e Monetize APIs to generate revenue—Advanced API man- 
agement will provide the functionality needed to package, 
price, and sell data products or services via any com- 
bination of free, freemium, purchase, subscription, or 
consumption-based models. It should also simplify inte- 
gration with analytics and billing services. 


e Build digital ecosystems to enhance business value—This 
involves providing granular control, compliance, security, 
and reporting mechanisms needed to support the expan- 
sion of digital value chains across a wide range of plat- 
forms, apps, devices, partners, and third parties. 


DevOps for Digital Leaders 


e Create efficiencies through analytics and optimization—By 
providing instrumentation and analytics that allow them 
to optimize technical and business performance, API 
management will encourage developers to build more 
efficient, performant, and scalable digital ecosystems. 


API Management: Essential Integrations 


When considering API tooling, it’s important to seek out integrated capabili- 
ties that help establish information feedback loops that cross-functional teams 
need to drive API improvements. 


API Performance Monitoring 


API performance monitoring provides a huge opportunity to increase quality, 
but has proved challenging. This is primarily because development and opera- 
tions teams use two classes of tools to address specific performance issues. 


API Management Gateways offer tremendous detailed insight into APIs from 
the client-side perspective (e.g., API throttling, policy, and routing), but lack 
the depth of analysis offered by Application Performance Management (APM) 
solutions—especially clear insight into the impact of API performance on 
critical business services. 


Application Performance Management (APM) solutions provide powerful analysis 
of application performance (e.g., transaction tracing and differential analytics), 
but lack inclusion of APl-level data for related monitoring and root-cause 
analysis, as well as specifics on the gateway itself. 


As a result, many problems can arise: 


e An operations team receives calls that API gateway ser- 
vices are underperforming, but this is reactive and after 
the fact because they’ve received no alerts in APM that 
indicate latency issues. 


e An application support team is called to field problems with 
mobile transactions using an API gateway, but because 
APM visibility ends at the firewall or proxy and they can't 
see the actual gateway services themselves, they struggle 
to identify the root cause. 


e An API development team has gateway insight, but because 
they lack visibility into back-end application performance, 
they cannot easily determine overall service impact. 


Bridging the gap between API management and APM solutions via a dedicated 
integration point (see Figure 4-3) can help teams collaborate effectively. 
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API Performance Monitoring 


Real-time , in-depth and customizable visibility into critical 
API-level performance data. 


Figure 4-3. API performance monitoring extends the functionality of API management and 
APM solutions 


By monitoring detailed API metrics and data traversing gateways in combination 
with APM analytics, teams can detect slow growing problems and fast acting 
acute ones. This allows faster response to problematic API service latency 
conditions before customers are impacted. 


When including detailed analytics at the API layer (e.g., front-end latency, pol- 
icy violations, and routing failures), root cause analysis becomes more thor- 
ough and conclusive. 


Through real-time and historical analysis of API performance information, 
developers and operations have a clear understanding on what API design 
improvements are needed. 


Note Integration between API gateways and APM solutions is especially valuable in pre-production 
testing. With complete end-to-end API performance visibility before a system goes live, teams are better 
positioned to remediate complex performance issues before they impact the business. 


API Development and Testing 


A 2015 Freeform Dynamics report indicated that it is 2.8 times more likely 
that digital disrupters are utilizing APIs in development? However, teams 
remain constrained by circumstances impeding success. In a Voke report, for 


?Freeform Dynamics, Exploiting the Software Advantage: Lessons from Digital Disrupters, October 
2015: http://transform.ca.com/rs/117-QWV-692/images/457955-Exploiting-the- 
Software-Advantage-2015.pdf 
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example, 56 percent of respondents stated that critical resources were not 
available when development and testing teams needed them.? 


API management solutions simplify the process of developers gaining access 
to APIs, but if back-end systems aren't available for testing connectivity, there 
can be delivery delays. 


An elegant solution to this problem is the integration API management with 
service virtualization solutions. Described in more detail Chapter 5, service 
virtualization emulates the behavior of dependent systems; enabling develop- 
ment and testing to be conducted free of constraints. 


Service virtualization is a great pairing for API management. It allows develop- 
ers access to a developer portal where they can quickly discover, learn to use, 
and integrate APIs into their apps. Then, with virtualized services, these apps 
can be tested against virtual back-ends, loads, latency, etc., to validate that apps 
under development apps will perform optimally in a variety of conditions. 


When combining service virtualization with API management, teams should 
seek out integrated capabilities that all allow developers and testers to: 


e Secure access to a directory of virtual services, enabling 
faster service discovery and consumption. 


e Switch automatically between real and virtualized envi- 
ronments, improving development velocity, increasing 
contract and data fidelity, and lowering defect counts. 


e Faster creation and easier maintenance of virtual services 
by leveraging API management data. 


Note According to the aforementioned Voke report, simultaneously accessing APIs and virtual 
services accelerates application development by 97 percent. 


Virtualizing the behavior of APIs and services and adding API management 
makes it easy for developers to access APIs that have been published and 
secured. As illustrated in Figure 4-4, working prototype APIs can be published 
and connected to virtualized back-end web services, so developers can begin 
using them as soon as they are available, rather than having to wait until the 
process is complete. This also allows teams to operate in parallel, where dif- 
ferent development teams can work on different features at the same time. 


3Market Snapshot™ Report: Service Virtualization by Voke Research: https://www. 
ca.com/au/collateral/industry-analyst-report/voke-market-snapshot- 
report-service-virtualization-iar.register.html 
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Figure 4-4. Eliminating API development constraints to enable parallel work streams 


Taking a Strategic Approach 


To fully embrace the APIs opportunity, organizations will need more than just 
tools, they will need a strategy that will help engender an understanding that 
APIs are more than just a technology; they are a digital enabler. 


Program managers (or API owners) need to be responsible for creating this 
strategy, communicating it to executive-level decision makers as well as the 
architects and developers on the front line. 


The first step should involve establishing a clear business objective and a vision 
statement, which needs to be aligned with the company's overall mission and 
goals. Making this link is often quite easy. 


For example, a retailer's vision might be, “Give our customers a compelling 
shopping experience while offering the best possible quality, service, and 
value". While the API vision might be, “Leverage APls to build a high-quality 
and engaging shopping experience that seamlessly crosses all of customer 
engagement channels." 


The next step should be to build a business model around the API vision, 
outlining the details of: 


e Costs, resources, and efficiencies—The systems, relation- 
ships, activities, and other resources the program wwill 
leverage and how the program will empower the enter- 
prise to deliver on its strategic goals. 


e Value, revenue, and innovation—The customers, markets, 
and channels the program will target and how technical 
innovation will make it possible to generate new revenue. 
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e Operational processes—The tools and approaches needed 
to effectively control, measure, optimize, and deploy a 
large number of APIs throughout the development and 
operations lifecycle. 


At the core of this business model, there should be a value proposition that 
clearly outlines the real, measureable outcomes that the API program will 
deliver to the business. 


API owners or product managers should also engage the business when defin- 
ing the API vision and delivery model. This will ensure: 


e Alignment with core business processes 
e Business value realization 
e Effective measurement criteria. 


Taking a strategic approach to APIs will enable organizations to overcome 
existing challenges around application integration, data silos, and fragmented 
omni-channel experiences. 


Building an API Future, Faster 


Digital disruptors have already recognized the value that APIs and effective 
management can deliver in terms of customer engagement, satisfaction, and 
retention. They're already using APIs both internally and externally: more than 
65 percent of digital disruptors use APIs internally for building their web appli- 
cations, mobile apps, and back office systems and externally for integrating 
third-party services or enabling external developer access to their systems.* 


Having the right management strategy and tooling can help organizations 
advance their API business goals by increasing maturity levels (see Figure 4-5). 


e Digital services—Organizations develop ad-hoc APIs that 
digitize simple transactions and make them available and 
secure for third-party use. These APIs bring traditional 
services to the Web and mobile devices. 


e Partner integrations—Organizations expand their business 
model by developing a top-tier API with numerous capa- 
bilities, and integrate it with select and trusted partners. 
While the API itself enables the connection, the requisite 
SLAs, agreements, documentation, and support have to 
be managed "offline". 


^Freeform Dynamics, Exploiting the Software Advantage: Lessons from Digital Disrupters, 
October 2015 
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e Partner Infrastructure—After success with some integra- 
tions, organizations look to roll them out to a larger com- 
munity of partners via an API portal or marketplace. They 
offer base capabilities for anyone who wants to consume 
them or create entirely new use cases. Here they not 
only provide the back-end for existing user experiences, 
but infrastructure for new ones as well, with standardized 
processes around SLAs and support. 


e API ecosystem—As organizations continue to monetize 
APIs and learn, they make deeper investments into build- 
ing innovative, new capabilities that create competitive 
advantage. Here, APIs push the limits into a broad eco- 
system, creating entirely new sources of revenue and 
enabling the adoption of entirely new technologies. 


HIGH 


Partner 


Digital Services Infrastructure 


Ecosystem 


Figure 4-5. API maturity Levels 


Summary 


Effectively managing APls is a critical business differentiator. As Mohan 
Subramanian, associate professor of strategy at Boston College's Carroll 
School of Management, confirmed: "APIs provide the key to unlock new 
growth opportunities at an unprecedented scale in our digitally connected 
economy. The capability to manage APIs will soon become a primary driver of 
competitive advantage.” 


Having the right API management tools and processes in place, organizations 
will be able to build the connections, digital services, and user experiences 
they need to succeed. 


*The Chief Digital Offer's Guide to Digital Transformation: https://www.ca.com/ 
au/collateral/ebook/the-chief-digital-officers-guide-to-digital- 
transformation.register.html 
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As discussed in this chapter, this requires taking managing the entire API life- 
cycle so that organizations can consistently deploy, measure, and optimize the 
growing volume and variety of APIs needed to support digital transformation 
initiatives. 

In the next chapter, we'll continue our exploration of the DevOps-enabled 
software factory by examining all the testing constraints facing practitioners 
today. We'll also describe some essential automated testing strategies needed 


to ensure that optimal quality and compliance is consistently achieved in the 
most cost-effective way possible. 
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Continuously Championing Quality 


The volume and velocity of software innovation afforded by DevOps is per- 
haps the biggest driver of IT’s shift to this new method of delivery. But when 
organizations neglect quality in the head-long rush to DevOps glory, the glass 
can only ever be half empty. 


The business-technology landscape is littered with many examples of what 
happens when software speed has been pursued at the expense of qual- 
ity. Perhaps the most extreme is Knight Trading, where a software update 
accessed outdated code (8 years old) that made more than $440 million in 
bad trades in less than 30 minutes.'? 


DevOps principles and practices are therefore not only intended to improve 
the tempo of software releases but also increase quality—and as with delivery, 
this must happen continuously! 


However, with DevOps' focus on an automating the software pipeline, it's 
clear that traditional methods for ensuring quality must now be questioned 
and reviewed. Separate teams working in silos, working with centralized pol- 
luted test data, and performing manual tasks late in the software development 
cycle is no way to sustain quality. 


'http://www.bloomberg.com/news/articles/2012-08-02/knight- shows - 
how-to-lose-440-million-in-30-minutes 
?https://www.sec.gov/litigation/admin/2013/34-70694.pdf 


© CA 2016 
A. Ravichandran et al., DevOps for Digital Leaders, DOI 10.1007/978-1-4842-1842-6 5 
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Progressive businesses understand implicitly the connection between speed 
and quality; increasing the cadence of releases, yes, but championing qual- 
ity through the application of advanced DevOps automation. One illustrative 
example is AutoTrader.com, the online marketplace for car buyers and sellers. 


Case Study: Auto Trader.com 


Every month more than 18 million people use AutoTrader.com to search for 
a used car. The site does more than host advertising for sellers; it also helps 
people research and compare cars and trade in their old vehicle.? 


Based in Atlanta, Georgia, AutoTrader’s goal is to make exchanging vehicles 
simpler and more secure than ever before, while maximizing value for private 
and trade buyers and sellers. 


When AutoTrader’s services were mainly available through a browser, it saw 
itself as a media company. The emergence of smart mobile devices and the 
application economy has changed all that. 


"Right now we're a technology company. The technology aspect is very 
important to AutoTrader as a business,” reveals Adam Mills, Senior Manager 
of Application Development at the company. 


Ten years ago AutoTrader released just four web services updates a year; 
today it expects to release one almost weekly. As Mills explains, “We have 
to keep up with changes to current operating systems and devices as well as 
evolve our own offerings. Customers expect us to deliver great new function- 
ality in weeks rather than months.” 


Customers also expect the same excellent experience whether they are 
accessing AutoTrader via an app or a browser on a mobile, desktop, or laptop. 


In a highly competitive market, this excellent experience is a key differentia- 
tor for AutoTrader. “Our app has to be the best,” explains Mills. “All it takes 
is a couple of bad customer experiences then everybody's talking about it on 
Twitter and we lose market share.” 


As the functionality of AutoTrader.com and the number and variety of devices 
it supported grew, software testing became complex, costly, and time-con- 
suming."VVe had to set up huge emulation environments, buy the licenses, and 
ensure all the services were talking to each other,” recalls Mills."But because 
there were so many interdependencies, we couldn't complete all the different 
tests in the same timeframe. We then had to find all the physical devices, plug 
them in, and test our code on every one.” 


Full Story: http://www. ca. com/content/dam/ca/us/files/case-studies/autotrader- 
avoids-300000-in-testing-costs-with-ca-service-virtualization.pdf 
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With plans already underway to adopt a DevOps approach to bring together 
its disparate development teams, AutoTrader realized that virtualizing differ- 
ent services would enable even greater unity. 


AutoTrader searched the market for a service virtualization solution that sup- 
ported the DevOps approach, inviting a select number of vendors to demon- 
strate the solutions. 


After implementing a solution in six weeks, AutoTrader began using the solu- 
tion to simulate apps behaving normally and performance issues. “Teams can 
test how resilient their services are and answer those key ‘what if’ questions, 
like ‘what happens if the database crashes?” comments Mills. 


Mills envisages that soon the last human interaction with a piece of code will 
be when a developer checks it in. Test, build, and deployment will be auto- 
mated, reducing processes that previously took weeks to just minutes. 


AutoTrader has been able to accelerate testing, while improving quality and 
freeing up resources. As Mills confirms," The solution means we can complete 
testing in hours rather than weeks. Previously we would have needed hun- 
dreds of testers to check performance on every device, but now we can test 
all devices automatically while our team focuses on higher value activities." 


The time taken to set up a new testing environment has also been cut from 
two weeks to two days, with costs dramatically reduced. AutoTrader.com has 
been able to: 


e Cut integration time from three days to three hours 


e Save and average of 567 man-hours—or 2.5 people—per 
release 


e Avoid $300,000 in test hardware and software costs 
e Decrease software defects by 25 percent 


As Mills concludes, “By getting new releases and services out the door quickly, 
we can provide a better experience to millions of car buyers and sellers and 
continue to differentiate in a competitive market.” 


Testing Times 


Apart from illustrating the importance of software quality, the AutoTrader 
story shows that this doesn’t have to slow things down. As the forward-think- 
ing Mills suggests, automation will be key for testing to become established 
within both continuous integration and continuous delivery processes. 


Testing is essential to DevOps because it brings the discipline smack-bang into 
the development processes and avoids the problems (e.g., release delays and 
quality issues) created by leaving QA as a gate or rubber-stamp function only 
performed at the very end of the cycle. 
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This isn't to say that the role of tester will be subsumed with development, 
but the discipline will change. Rather than providing a transactional service 
to developers (e.g., executing tests and handballing the bad news), the focus 
of testing will shift toward a more consultative role that will help developers 
learn how to write better tests and improve their approaches to scanning for 
quality. Developers aren't necessarily hard-wired to look for quality issues, and 
even though the vast majority do care about quality code, they are still going 
to miss issues and opportunities for improvement. 


Tip To establish testing expertise DevOps style, leaders should consider positioning their 
teams in a way that can add the most value across the software development lifecycle. This may 
involve embedding specialists within agile product teams or even creating a center of excellence. 


As advanced automation becomes more pervasive, QA and testing profes- 
sionals will need to become better skilled at fully leveraging it. This involves 
providing a comprehensive and elevated test discipline rather than just exe- 
cuting a series of day-to-day tasks. 


Some new skills include: 


e Thinking beyond pass or fail—Helping the business under- 
stand what the customer actually experiences and how 
that can be best simulated during testing. Essentially sup- 
plying the right data and real-world conditions needed to 
better support and enhance a quality experience. 


e Intimate understanding—With the complexity surrounding 
applications today, QA, and testing staff need to become 
far more proficient at understanding all the intricacies. At 
a minimum, this means visualizing all dependencies and 
being able to remove constraints. 


e Assurance to analytics—QA has traditionally been focused 
on documenting defects and reporting back to develop- 
ment. This must shift toward collecting and aggregating 
data from a broad range of automated tests to determine 
the actual cause of defects and where more rigorous 
testing is needed. 


e Early and thorough testing—With agile increasing the vol- 
ume of user stories, it makes perfect sense to incorpo- 
rate testing into the acceptance criteria. At this early 
stage any progression into the sprint should be depen- 
dent on reviews involving QA, but also security and IT 
operations too. 
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Note 


With agile and DevOps, quality is baked into the SDLC, not bolted on at the end. This 
requires establishing ownership at a cross-functional level, not devolving to one team. Automation 


to support this goal should be available to all stakeholders, not just QA/testing teams. 


Mentorship over conflict—Rather than constantly being 
called in to address fragile developer-written tests, 
DevOps focused QA will work closely with their coding 
colleagues to continuously improve testing resilience. 


Ambiguity to clarity—Vague requirements stored in multi- 
ple formats leads to defective software and a sub-optimal 
customer experience. Teams should seek out methods 
to map changing requirements to visual models and elimi- 
nate ambiguous requirements and the costly defects they 
create. 


Quality over quantity—Having many redundant, duplicate 
tests guarantees nothing but cost overruns and delays. 
QA and testing teams should consider advanced automa- 
tion methods that generate the smallest number of test 
cases needed for 100 percent functional coverage—all 
linked to the right data and expected results. 


Agile Testing Trifecta 


A key goal of DevOps should be making testing an accelerator, not an 
obstacle to fast application delivery with the highest levels of quality. To 
support this, more advanced testing tools are needed, equipping QA and 
testing teams with three essential capabilities needed to support agile 
and continuous delivery methods. This “testing trifecta,” as illustrated in 


Figure 5-1, includes: 


Test automation to create test cases right from requirements 
Generating synthetic test data to be used on demand 


Test constraint removal by virtualizing every environment 
that needs to be accessed 
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Figure 5-1. Testing trifecta for agile and DevOps 


Test Automation 


Today, nearly every company is in the software business. Although an orga- 
nization may sell a tangible product, their use of software to streamline and 
enhance the customer experience means they must place higher importance 
on quality application delivery. Frequently, applications that are rushed through 
the development cycle without adequate testing often encounter costly 
defects that impact the customer relationship. 


Just like building a house, the foundation is key to successful software con- 
struction. If the foundation has issues, there is a high likelihood for expensive 
delays further into the process. Using the right development tools at the onset 
will help ensure that the software foundation is properly defined, constructed, 
and tested while keeping quality and end user goals top of mind. 


Incomplete Requirements Equals Faulty Software 


Many quality problems eventuate during the requirements design phase. This 
is because software requirements are typically ambiguous, incomplete, and 
stored in many different formats by numerous people within the organization. 
Test cases are then manually defined from incomplete requirements and thus 
the stage is set for foundational cracks to appear even before the application 
has been built. 
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Further, the manual definition of test cases is a slow and unsystematic process 
that leads to perhaps 10-20 percent functional test coverage. Testers end up 
testing the same features over and over again without knowing for certain the 
results. As a consequence, defects are detected later much in the develop- 
ment cycle, leading to costly rework. 


An Automated and Agile Approach 


If testing is going to keep pace with continuous delivery goals, it needs to 
become much more automated and agile. Adopting a requirements-driven 
(or customer centric) approach is the first step and may require software 
solutions to force the change. With the advanced tools, testers can generate 
the right test cases needed for maximum coverage. Test assets can be derived 
directly from the design and updated automatically to reflect changing user 
needs. 


Tools in this category allow user stories to be imported and modeled as an 
active flowchart (see Figure 5-2). 


Figure 5-2. Agile requirements design allows user stories to be verified with end users 


Active flowcharting helps eliminate requirements ambiguity and reduce 
defects early in the design phase. This class of tool will also generate the 
smallest set of automated tests needed for maximum coverage. Importantly, 
and to support the drive to testing as a discipline becoming much more 
proactive, these tools also help testing teams know which features should 
receive the most rigorous testing based on analytics and metrics gathering 
capabilities. 
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Achieving Complete Test Coverage 


As applications become more complex and distributed, business logic is no 
longer found only in the user interface (UI) and the database (as with cli- 
ent/server models), but extends across multiple tiers and technologies. This 
becomes further complicated when applications consume underlying services 
from cloud providers or third-parties, or use highly interactive presentation 
layer technologies. 


Organizations are also implementing more agile development methods from 
distributed teams, yet the use of shareable, reusable test assets between these 
teams is limited or non-existent. Traditional tools designed for more linear 
style waterfall development are often employed, but lack extensibility, only 
supporting the needs of one group. For example, code-based unit testing tools 
for developers that are unusable by QA and functional user interface (Ul); fail- 
ing to translate errors into repeatable defect identification needed by devel- 
opers to catch bugs earlier. 


This requires a much higher degree of test automation and collaboration 
among stakeholders. As testing efficiency and effectiveness become para- 
mount, a new continuous testing model supported by advanced automation 
technologies should be the goal. Only through this coordinated approach can 
organizations build the scale needed to meet future demands. 


Meeting these goals can only be ensured when every layer of the applica- 
tion and the complex interactions between components is automatically 
tested and verified throughout the software lifecycle. This involves provid- 
ing complete test coverage with the ability to invoke and verify the behavior 
of each component, singularly or as an end-to-end service. Solutions in this 
class must therefore provide industry-leading standards support, with native 
integration to J2EE servers, integration suites, and ESBs. To help strengthen 
the DevOps toolchain, solutions will also integrate popular open source tools 
(e.g., Selenium Builder for UI testing), thereby enabling end-to-end testing 
from user interface all the way to back-end systems. 


Case in Point: Mobile Testing 


True extensibility means one tool coordinating and running functional tests, 
test Uls, and APls on multiple mobile devices under various conditions. 
Traditional approaches to testing fall down in the mobile world because it's no 
longer sufficient to just to test the "function" of the application. Code needs 
to be tested using the same conditions that the app will run under when in the 
hands of a user, with experience-based metrics and test reports reviewable by 
both the user acceptance teams and development to further improve quality 
and expedite defect resolution. 
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To support the goal of ensuring high-quality during continuous integration 
(critical for mobile apps where changes updates occur more frequently), such 
solutions should provide unattended automation coverage. This involves exe- 
cuting tests against real mobile devices connected locally or from the cloud 
(see Figure 5-3) immediately code is committed. 


Virtualized services described in more detail later in this chapter address the 
common mobile testing challenge of testers needing access to dependent 
systems for end-to-end analysis. Tests should also allow for different profiles 
simulating network conditions, location, background applications, and device 
orientation. This way teams can report and benchmark the user experience 
of different personas at various points in the app workflow. 
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Figure 5-3. Automated mobile tests on smartphones, using multiple OS versions on multiple 
carrier networks and in different locations worldwide 


Tip When mobile apps are in full production, consider using app experience analytics tools for 
continued insight into both usage and performance. Results can be valuable for determining where 
functional and performance improvements are needed. 


Test Data Management 


The second part of the testing trifecta and an area rife for improvement in 
the software development lifecycle is in test data management. Every tester 
needs quality test data and quickly. The challenge is getting the right data to 


717 


78 


Chapter 5 | Test 


match their tests when they need it. As companies have improved their devel- 
opment processes, moving from Waterfall to agile, testing has lagged behind. 
Again, manual processes cannot keep pace with a company's test data need, 
with companies relying heavily on teams of people constantly creating and 
maintaining test data. 


Another major challenge when managing test data is ensuring compliance 
with legal and regulatory requirements. Many organizations apply the neces- 
sary rigor when protecting personal and sensitive customer information in 
production, but neglect to consider the implications when working with data 
in non-production environments. In the event of non-compliance, this can 
mean significant consequences, not the least brand reputation, but also finan- 
cial loss due to fines and penalties. 


Many industry-specific regulations come with their own unique sets of test 
data challenges, and some introduce new complexities. Take the General Data 
Protection Regulation (GDPR) for example. The GDPR is designed to protect 
the rights of European Union (EU) citizens where the processing of their per- 
sonal data is concerned. 


Although many companies will have already adopted privacy processes 
and procedures consistent with the directive, the GDPR contains a num- 
ber of new protections for EU data subjects and threatens significant 
fines and penalties for non-compliance (up to 4 percent of annual global 
turnover or 20m euros, whichever is greater) once it comes into force 
in May 2018. 


GDPR introduces many new obligations in areas such as data anonymization, 
breach notification, and trans-border data transfers, to name just a few. Many 
have implications for test data management. One example is the "right to 
erase,’ where individuals may notify businesses processing their data what 
they may or may not use that data for, including testing. 


Complying with obligations like this carries a huge overhead. If customers 
state they don't want their data used (even it is masked), then testers will 
need to acquire subsets of data and apply filtering rules. They will also need 
to ensure their methods can track every record not approved for testing and 
be fully auditable. 


Many businesses might pursue programmatic solutions to these test data 
problems, but this only increases the development burden and poten- 
tially introduces additional fragility. One alternative, of course, is to use 
more modern synthetic test data generation to avoid these problems 
completely. 
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Facets of a Gold Standard Solution 


To address the complex issues involved with the acquisition of quality data and 
regulatory compliance, modern test data management solutions will provide: 


e Synthetic test data generation—Synthetic data contains all 
of the characteristics of production but none of the sen- 
sitive content. This ensures teams are provisioned with 
secure, realistic data that maintains referential integrity as 
part of a move toward a "Live Data Exclusion" model for 
testing. In addition to addressing compliance issues, and 
as illustrated in Table 5-1, synthetic test data generation 
can address other constraints. 


Table 5-1. Removing Constraints with Synthetic Test Data 


Constraint Resolution 


Regulations and compliance Lower risk as data is generated 


Data functional coverage Measure and get 100 percent coverage 


Capacity to identify data "holes" by comparison between 
environments or directly identify data from test cases 


Test database size Only stores most efficient set of test data 


Provisioning delays Provisioning in minutes, on-demand, through a web portal 


Capacity to book data for each tester 


e Scalable end-to-end platform—Tools should clone subsets 
of data into target environments and be capable of secur- 
ing millions of data rows in minutes using automated data 
profiling and advanced masking engines. 


e Test data allocation—Tools must facilitate automated data 
discovery for testers to receive exact datasets, linked to 
their test cases. 


e Test data warehouse—The ability to store pools of test 
data as reusable assets in a central repository and test 
multiple versions and releases in parallel. 


The following checklist can also be useful in assessing the efficacy of test data 
management solutions: 


e Provides a standard set of data to test 


e |s"production-like" 
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e Covers all possible tests that need to be run, including 
future and negative scenarios 


e Contains just enough data to test repeatedly 


e İs up-to-date, while also containing and supporting all 
previous data 


e Contains absolutely no sensitive data 


Combining with Test Automation 


The test automation methods described in the first part of this section, espe- 
cially the ability to create test cases right from requirements, are powerful 
capabilities in their own right. However, combine them with test data manage- 
ment and testers can move beyond just executing tests to proactively driving 
quality improvements. 


By way of example, consider a two-way integration between test data man- 
agement and agile requirements definition. Here, test matching functionality 
should be available to locate or create the data needed to execute the opti- 
mized test that has been built straight from requirements. The test data itself 
would be stored in a central test data warehouse where it can be provisioned 
on demand and used in parallel with development efforts. 


Through dynamic building, testers can request the data they need based on 
specific criteria and receive it in minutes from a self-service web portal. The 
provisioned data is cloned and version controls are applied to update data to 
immediately reflect any changes in requirements. 


Using the integrated approach, teams benefit in many ways: 


e Distributed test teams can work with multiple application 
versions with matching test data 


e Automatically locate or create test data based on specific 
testing needs 


e Test for outliers, unexpected results, and negative 
scenarios 


e Significantly reduce the time and resources required to 
provision test data 


e Generate synthetic data (data from scratch) without the 
need to mask production data 


e Create test data quickly for use in service virtualization 
to speed testing and increase quality; feeding data directly 
to service virtualization engines and linking test data with 
virtual end-points 
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Implementing a test data management strategy is crucial to realizing the goal of 
continuous application delivery. By making test data accessible during require- 
ments design, teams can streamline and eliminate the bottlenecks associated 
with test case creation and locating the right test data. 


Test Constraint Removal 


There is a fundamental shift in the way enterprises build applications today. 
In the early days of mainframe and client/server applications, you had a much 
more limited scope of applications—all of the components from the database 
to the Ul could be under one development and testing team's control. 


After the Dot-Com days of the early 2000s, a new style of composite applica- 
tions arose. The new approach to developing software, including agile, created 
two new challenges for organizations 


e Constraints created by the highly parallel development efforts 


e Dependencies on consistent behavior of the components 
in the system 


These complications increased the complexity and cost of developing and 
maintaining composite applications. 


Applications today are the result of many decades of "building systems on 
top of systems," which creates huge chains of dependencies. These complex 
architectures mean software development is more difficult, more costly, and 
more complex than ever before. 


Many large organizations now find that many of systems they depend on such 
as mainframes, databases, and external services are constrained and not acces- 
sible by developers and testers when they are most needed. 


For instance, a needed mainframe may be off-limits, a system of record could 
have bad data, or a third-party service may still be under development. 
Attempts to reproduce these environments—by manually coding stubs and 
managing test data—are costly and inconsistent. 


One customer with constraint issues put it this way, “I can’t do anything until 
| have everything... and | never have everything!" 


In a recent Voke Market Snapshot Report on Service Virtualization (January 
2015), over 500 companies validated that constraints are a major hurdle to 
innovation in the software development lifecycle.* The report mentions that: 


e 80 percent of teams experience delays in development 
due to constraints everywhere across the SDLC 


^https://www.ca.com/au/collateral/industry-analyst-report/voke-market- 
snapshot-report-service-virtualization-iar.register.html 
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e 56 percent of critical dependencies are unavailable when 
development and test need them 


e 70 percent of teams face prohibitive restrictions (delays, 
time, and fees) when needing to access third-party systems 


Service virtualization solutions can solve these constraint issues by capturing 
and modeling dependent systems. As virtual versions of the real thing, these 
services simulate the constrained components in any environment, providing 
low-cost, 24/7 available models. 


When developers and testers use service virtualization, the services behave 
and perform similar to the real thing, but without the underlying hardware 
and software complexity of a physical system. Development and testing con- 
tinue just as they always have, but less constrained, and without contention 
between teams for environments, labs, test data, and so on. 


Although service virtualization solves many different development problems, 
four common ones are seen repeatedly: 


e “Shift left” —Enabling parallel software development, test- 
ing, and validation for faster time-to-value with earlier 
defect resolution (see Figure 5-3) 


e Infrastructure availability—Eliminating much of the con- 
current demand for environments and hardware that 
agile development creates 


e Performance readiness or solving the challenging prob- 
lems of properly evaluating the scalability of applications 


e Scenario and data management—Often eliminating the 
need for complex test data management, system setup, 
and other complexities 


Dev Dev Dev Dev Dev Dev 


3 2 3 Integration 


With Service i irtualization quality 
effort movefi earlier in lifecycle 


Dev3 Reduction in SDLC 


» Integration 


Figure 5-3. “Shift-left” testing with service virtualization 
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One common problem that service virtualization solves is in the area of inte- 
gration. Customers buy companies, they provide service to third parties, or 
they are updating applications for functionality, compliance, or architecture. 
Each of these challenges presents an opportunity for service virtualization to 
improve the software development process. Integration teams and customers 
have the resources they need for software development and testing, without 
the added expense of acquiring additional hardware and software. 


When time-to-market matters, service virtualization offers an excellent 
opportunity to shorten development lifecycles. Service virtualization reduces 
the constraints of software development, allowing more teams to effectively 
work in parallel, without underlying dependencies. Typically, service virtualiza- 
tion users experience a 25-50 percent reduction in release times. 


A huge opportunity for service virtualization is in the area of performance 
engineering. Creating a lab capable of handling and testing to production 
capacity loads is difficult and resource intensive. Furthermore, ready access 
to systems such as mainframes and transaction servers may be impossible. All 
this makes performance testing expensive, unreliable, and inconsistent. Some 
teams might have a small window for testing, while others will have to wait for 
an entire application architecture to be assembled before any testing can com- 
mence. But by using service virtualization, teams can performance test each 
individual component, identifying many performance problems earlier in the 
lifecycle, and reduce, even eliminate, the amount of final performance testing 
needed in a production-like lab. 


Using traditional Waterfall methodologies for developing software, much of 
the activity of development and testing of the application happens in a series 
of steps, one after another. But by eliminating constraints common in typi- 
cal software development practices, service virtualization enables much of 
the SDLC to operate in parallel and the steps within it become less time 
consuming. 


Using service virtualization, developers can have their own private environ- 
ments for coding, directly from the laptop. They don't share environments and 
don't need to wait for other developers to finish their work. 


With service virtualization, much of the testing at a component level can 
“shift left,’ or be moved earlier in the SDLC. Because each component can 
be tested individually (instead of waiting for a complete assembly), unit and 
regression testing happens sooner and is more complete, and defects are 
identified long before integration or user acceptance testing. Finding defects 
earlier means developers fix issues at the point in time they incur the lowest 
cost. This avoids defects leaking into later stages or even into production and 
become harder to resolve because developers have been moved onto other 
projects. 
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As teams increase service virtualization maturity, regression and individual 
component testing become increasingly automated. Now validation as early as 
code check-in is possible, making defect detection a consistent and repeatable 
process. Again this is possible because service virtualization allows compo- 
nent level testing in isolation, without underlying dependencies. 


Once automation is implemented, you can easily make it a continuous process. 
Using this approach, any change breaking interfaces, contracts, or use patterns 
are easily detected before the code disrupts other services or applications. 


The deployment of service virtualization at one large bank solved two critical 
challenges. First, by eliminating system dependencies, testing began far earlier 
in the development cycle. Defects in code no longer lurked until UAT, but 
were found much earlier. 


Additionally, the bank’s formally serial processes were set in parallel, dramati- 
cally reducing release times. 


Using virtual services reduces the demand for physical hardware and test 
labs. This approach is distinctly different, and complementary to, hardware 
virtualization. With service virtualization, you virtualizes services and business 
functionality instead of hardware. 


When demands for hardware decrease, so do costs. The challenges and costs 
of provisioning labs and equipment, software and configurations disappear. 
The physical hardware demand decreases dramatically, freeing budgets for 
application and business investments instead of capital assets. Demand for 
data center rack space, power, and storage also decrease. 


In performance testing, service virtualization helps customers reduce cost and 
increase quality and flexibility in several ways. Customers can load test at the 
component level. Instead of waiting until the application is complete, compo- 
nents are tested for volume and capacity independently, locating bottlenecks 
and issues early. 


Production-only systems such as master databases, mainframes, and third- 
party systems not normally available for load testing are virtualized, creating 
an always-ready, highly scalable virtual back-end immune to traditional load 
testing constraints. This benefits users both in convenience and by reducing 
the cost associated with replicating expensive back-end systems. 


Customers may also face third-party access or software license fees. Service 
virtualization eliminates the need for highly scalable versions of these systems 
by virtualizing their behavior. For organizations selling services, virtualized ver- 
sions of their entire platform are made available to customers in virtual form. 
Validating against a virtual back-end ensures production readiness without the 
complexities of full production-style dev/test implementations. 
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Using service virtualization addresses many thorny issues associated with test 
data management. For example, organizations struggled to set up just the 
right scenarios, only to "burn" them with a test cycle. Or, find it difficult to 
construct test scenarios for edge conditions and business logic. In such cases, 
it often becomes more expensive to set up the test harness than do the test! 


Virtualizing behaviors such as edge conditions, negative test scenarios, and 
error handling are easily configured in the behavior of the virtual service and 
are never "burned" since the virtual service is simply playing back behavior 
responses. 


With service virtualization, “test data" and scenarios are easily versioned and 
changed for each new requirement. In addition, when two test cycles or teams 
have differing needs for test data, they will not collide in the test lab. 


Summary 


With the advent of agile development, testing as a discipline is changing radi- 
cally. Using the approaches described in this chapter, testing can move beyond 
being a separate siloed function employed at the end of cycles, to becoming 
a more proactive, continuous, and analytical discipline that firmly establishes 
quality, whatever the pace of the delivery. 


While achieving this goal may require changes in mindset and organizational 
structure, what's indisputable is the need to adopt a comprehensive testing 
approach to address end-to-end automation needs, manage test data, and 
remove all constraints. 


In the next chapter, we'll examine the software releases strategies organiza- 
tions should consider as they move to a more continuous method of delivery. 
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6 


Deploy 
Building an Agile, Resilient, and Scalable 
Continuous Delivery Pipeline 


To keep pace with demands for new features and application updates, digital 
transformation must be driven by continuous delivery—the ability to rapidly 
and reliably release software across the pipeline at any time. 


By almost every metric, companies that address this imperative create a 
competitive advantage over those that lag. Yet few companies have actually 
developed the process maturity and scalable automation needed to deliver 
applications at the volume, velocity, and quality levels now required to remain 
competitive. 


Before looking the challenges and strategies needed to increase maturity, let's 
examine how advanced DevOps thinking backed by release automation has 
helped technology giant Citrix significantly cut deployment times and reduce 
errors during the release process.' 


'Full story:http://www.ca.com/content/dam/ca/us/files/case-studies/citrix- 
boosts-business-agility-and-accelerates-devops-adoption-with-ca-release- 
automation.pdf 


© CA 2016 
A. Ravichandran et al., DevOps for Digital Leaders, DOI 10.1007/978-1-4842-1842-6 6 
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Case Study: Citrix 


Citrix provides a range of virtualization, networking, and cloud solutions to 
around 400,000 customers worldwide. To help create more productive work- 
spaces both for its customers and own users, Citrix is continually looking at 
ways to improve its business operations and enable innovation. As part of this 
drive, Citrix created an Office of IT Delivery Optimization in December 2014, 
which is tasked with evaluating and improving all aspects of IT. As part of its 
optimization efforts, the team is embracing DevOps principles. As Eugene 
Lehenbauer, Worldwide IT Delivery Optimization Group Manager at Citrix, 
explains, “Adopting a DevOps approach to IT delivery will help us achieve bet- 
ter cross-team collaboration, faster delivery, and greater quality.” 


Although individual development, architecture, and design teams at Citrix had 
already embarked on their own DevOps journeys, the company wanted to 
take a more centralized approach to maximize results and share best prac- 
tices. "Supporting innovation and free-thinking is really important at Citrix, 
so we didn't want to impose a specific toolset,’ says Lehenbauer. “We did, 
however, want to give teams the option of using a proven enterprise platform 
for automating application deployments to help free up their people from 
repetitive and mundane tasks.” 


During the proof of concept exercise, Citrix moved from manual release 
processes to fully automated release processes, reducing deployment time 
by 80 percent. But that was not enough for the Citrix team. Development 
was inspired to re-architect the one large "MyCitrix" application into many 
smaller pieces, which, along with release automation, enabled the application 
deployment time to be reduced further to 94 percent. 


"Achieving such impressive and immediate quantifiable results has really 
helped accelerate the adoption of DevOps principles across the business and 
inspired development innovation," explains Lehenbauer. 


A central dashboard permits everyone involved to view the status of all 
releases, giving teams the information they need to act quickly and providing 
an audit trail for development and operational teams alike. 


The weekly updates to MyCitrix are managed via a release automation solu- 
tion. "Since deploying Release Automation, we've achieved faster delivery 
times and fewer issues. As a result, we now have more developers focused on 
innovating, rather than reacting,’ added Lehenbauer. 


Release automation has been a catalyst for Citrix's adoption of DevOps prin- 
ciples. It has enabled Citrix to take an enterprise-level approach to application 
delivery by automating application release tasks and orchestrating its continu- 
ous delivery toolchain. 
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Citrix has been able to significantly accelerate its application delivery and 
reduce the errors and time required during the release process. This has 
helped the company to be more responsive to customer needs, ensure com- 
pliance and auditability, and focus on innovation instead of repetitive tasks. 


Obstacles to Continuous Delivery 


As the Citrix story demonstrates, continuously delivering software is an 
extremely collaborative process that spans multiple departments, from devel- 
opment to test, to release management to operations. With so many stake- 
holders and motivations, the challenges faced by development-focused teams 
can be very different than those confronting operations professionals. 


Development Challenges 


With an emphasis on increasing throughput, major obstacles are delays and 
release bottlenecks. For example: 


e = Manual, time-consuming, error-prone environment provi- 
sioning and release processes 


e Numerous errors happening throughout the applica- 
tion release cycle and lots of detective work to find the 
source of problems 


e  [nefficiencies caused by the uncoordinated adoption of 
open source tools, leading to duplication of effort, redun- 
dant solutions, and disjointed integration 


e Slow response to customer feedback and market needs 
impacting customer retention and acquisition 


Operations Challenges 


With an emphasis on ensuring stability, major challenges involve guaranteeing 
resilience as the volume and velocity of deployments increases. For example: 


e Fractured release processes; managing with spreadsheets, 
scripts, and tools 


e Difficulty managing/tracking the volume of releases as 
more agile development ensues 


e Long weekends, low staff-morale and stress due to prob- 
lems when finally deploying to production 
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e  Double-digit application outages or downtime happening 
each month and needing an "all hands on deck" approach 
to resolve 


e Loss of customers and revenue due to downtime/outages 
or errors in application deployments 


Finding Common Ground 


Regardless of the issues facing each team, it's important that common ground 
and consensus is reached by tracking all issues preventing successful business 
outcomes. This is a shared exercise and involves all stakeholders collectively 
working to determine where the organization as a whole is on the path to 
automating software releases that drive a continuous flow of value to the 
business and its customers. 


Tip Consider organizing a continuous delivery "current state" workshop that brings all 
stakeholders together. These may include application owners, developers, enterprise architects, 
security managers, change managers, release managers, operations, and support. 


To facilitate open discussion, some good conversation-starters include: 


e In terms of continuous delivery, what are our agreed 
business goals and metrics? 


e How are we managing and executing application deploy- 
ments? What elements are heavily scripted and rely on 
manual intervention? 


e Howare we configuring environments from development 
through to production? Are different teams using differ- 
ent processes? 


e Across the software pipeline, what are the readily visible 
bottlenecks in the application release process? 


e What automation tools are currently leveraged (e.g., con- 
tinuous integration and configuration management)? Are 
teams using different tools? 


By jointly answering questions like these, teams can develop a structured 
understanding of where there may be weaknesses across the entire release 
pipeline and identify opportunities to automate and improve processes. 
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Tip Try not to restrict analysis to release teams and processes only. Look for opportunities 
where automation can help drive improvements in development and testing. Careful attention 
should be given to how the "current state" affects the work of others. For example, if there are 
release delays, how does this impact development? What processes are they using to circumvent? 


For example, are development teams being pulled off important refactoring work because of 
delays? Is testing being pushed late in the cycle or neglected because teams think they have time 
to do it later? 


Continuous Delivery Maturity 


As with all new technologies and best practices, organizations will be at differ- 
ent points on the journey to continuous delivery (see Figure 6-1). Some will 
have already begun, often by adopting facets of agile or even DevOps, while 
others will just be starting out. In fact, it's not uncommon for different teams 
across IT to be at different points in adoption. 
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Figure 6-1. Continuous delivery maturity levels 


Level |: (Manual) 


At this level, success depends on the competence and heroics of the people 
doing the delivery. Teams are very much operating in silos. Application releases 
are error-prone and infrequent and the business is badly positioned to act 
quickly on new opportunities, defend market position, or retain customers. 
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Level 2: (Scripting) 


Deployment processes are planned per release, and status is managed 
and tracked. Automation may exist for some deployment capabilities (e.g., 
scripts). Teams are probably using version control/repositories (e.g., Nexus) 
and doing automated builds using tools like Jenkins/CloudBees. It's likely 
that provisioning or configuration management tools like Chef or Puppet 
are used to help with delivery, but no application-centric end-to-end release 
orchestration is employed. 


Level 3: (Automated) 


Here, there are common, reusable, automated application delivery processes 
established across environments and releases. Release processes, release 
metadata, and release artifacts are monitored and tracked under full lifecycle 
control. Delivery automation exists at the environment release level, which 
may include leveraging existing provisioning and deployment automation capa- 
bilities (which may not be scalable). 


Level 4: (Continuous) 


At this stage, release automation orchestrates application release promotion, 
enabling predictable, monitored, and measurable continuous delivery from 
development to production. Organizations can deploy applications consis- 
tently across different types of environments and releasing software is a rou- 
tine and relatively low-risk event. 


Level 5: (Optimized) 


Now all elements are working in a fully orchestrated fashion to provide a 
zero-touch deployment—from planning to production. Continuous optimi- 
zation of end-to-end application delivery processes through feedback loops, 
with deployment patterns, scenario simulation, and analysis of operational 
release data are used to continuously improve cost-performance of applica- 
tion delivery. Teams manage multiple applications (multi-services) through the 
continuous delivery pipeline, which becomes a single point of control. There 
is also a strong focus on resilience and continuous availability. 


No matter where you sit on the continuous delivery maturity curve, one thing 
is clear—every new level provides tangible benefits. Processes become more 
automated and standardized. And teams become more productive, focusing 
on delivering differentiating features rather than managing unplanned work 
and maintenance tasks. They can handle the growing tempo and complexity 
of applications, while still ensuring quality and resilience. 


DevOps for Digital Leaders | 93 


Accelerating Maturity: Three Ways 


As Figure 6-1 illustrates, the adoption of automated release processes and 
tools typically drives major inflection points in a continuous delivery journey. 
There are three important considerations. 


The First Way: Connect End-to-End Release 
Management 
Scripting to Automated 


Taking an end-to-end release automation approach is essential in order to 
execute a successful continuous delivery strategy. Key to this is the ability 
to automate and standardize application releases all the way from develop- 
ment through to production, combined with capabilities to plan, manage, and 
optimize the release pipeline to improve quality and processes. Rather than 
act in isolation, release automation must easily integrate with other processes 
and tools (e.g., continuous integration, provisioning, and configuration man- 
agement) across the continuous delivery toolchain; seamlessly scaling as the 
volume, velocity, and complexity of applications grow. 


Taking this step to end-to-end release automation also supports DevOps 
adoption. It becomes easier for teams to have the release transparency, com- 
munication, and consistency needed for more purposeful collaboration. 


More importantly, cross-functional teams gain control and visibility of the 
entire release pipeline, looking at the release process systematically versus 
in silos. 


At this and any stage it's important to measure how improvements are help- 
ing support the business goals of continuous delivery that were identified 
before any toolset implementation. For a large Fortune 100 financial services 
company participating in a release automation ROI study, this involved increas- 
ing application release rates across the software lifecycle. As stated by the 
manager of DevOps enterprise release and deployment, "One of our core 
application deployments was done twice a week due to lack of automation, 
intensive manpower, and complicated deployment procedure. After automat- 
ing this application deployment with release automation, the application is 
being deployed at least 50 times in a week, all the way from continuous inte- 
gration to production.” 


?The Total Economic Impact?" of CA Release Automation, December 2015: http: // 
WWW. Ca.com/content/dam/ca/us/files/industry-analyst-report/the-total- 
economic-impact-of-ca-release-automation.pdf 
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Tip Never underestimate the people impact when introducing new automated release 
methods. Rather than enforcing enterprise adoption, consider small but important projects where 
benefits can be quickly demonstrated. This can become the catalyst for wider support. 


The Second Way: Operationalize Feedback Loops 
Automated to Continuous 


While automation is essential for continuous delivery, it's only the start of 
the journey. As automated end-to-end release processes become firmly 
entrenched, many new release challenges emerge. Taken individually or as a 
whole (as illustrated in Table 6-1), these pressure points drive a shift to better 
pipeline management. 


Table 6-1. Pressure Points Increase the Need for Advanced Release Automation 


Application Content Infusing releases with feedback more quickly 
Complexity Prioritizing deployment of the right content 
Demonstrating implementation against business requirements 
Preventing “polluted” content from reaching production 
The Pipeline Multiplier ^ Planning, tracking, and prioritizing many complex multi-level 
Effect applications and independently developed services 
Managing dependencies and avoiding conflicts 


Sharing resources between multiple teams, projects, and 


timelines 
Pipeline Tooling Juggling a growing breadth of open source, home-grown, and 
Expansion third-party commercial tools used across the enterprise by 


different teams 


As these pressure points intensify, organizations need to consider processes 
for executing multi-team, cross-app, composite releases, while ensuring 
all dependencies are handled. The proliferation of moving parts requires a 
"big picture" view of the pipeline to maintain throughput, contain issues, and 
ensure fast feedback. 


More importantly, the continuous delivery pipeline is becoming the single 
control point and application delivery is becoming streamlined, predictable, 
and risk-free. At this stage, release automation is orchestrating tools and 
processes beyond deployment, including application lifecycle management 
(ALM) and service management processes (e.g., change management). This is 
essential for DevOps, since it strengthens feedback loops and better informs 
decision-making. 
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Many more teams within in the enterprise should now be running apps 
through the single control point. If they are, they are better equipped to estab- 
lish a framework of continuous delivery best practice that's valuable across 
the organization. 


Note According to the 2016 State of DevOps report, high-performing IT organizations deploy 
200 times more frequently than low performers, with 2,555 times faster lead times.? 


The Third Way: Optimize the Continuous Delivery 
Pipeline 
Continuous to Optimized 


Although few departments are operating at this level, it is the pinnacle toward 
which all teams should aspire. 


With the continuous delivery pipeline being too important to fail, attention 
should become focused toward making the pipeline (so many teams depend 
upon) as efficient, stable, and resilient as possible. 


This involves shifting toward mastering the art of releasing multi-app, cross- 
app, multi-team applications and making deployments more predictable and 
efficient. Improving business execution through accelerated feedback loops 
will be another benefit, and by establishing a culture of continuous improve- 
ment, teams will embrace a "fail fast" culture and then apply the lessons 
learned within their release processes to prevent future problems. 


This notion of continuous improvement is well illustrated by a director of 
DevOps tools management at a leading Fortune 100 financial services com- 
pany, who stated, "Agile and continuous delivery can be nothing but a journey. 
You are never done; you are constantly moving the needle. There is always 
something you can do.”* 


Essential Toolchain Integrations 


While it's important to review functional aspects of release automation solu- 
tions, what's more important is examining a solution in terms of how it helps 
organizations increase continuous delivery maturity. 


32016 State of DevOps Report: https://puppet.com/resources/white-paper/2016- 
state-of-devops-report 

‘The Total Economic Impact?" of CA Release Automation , December 2015: http: // 
WWW. Ca.com/content/dam/ca/us/files/industry-analyst-report/the-total- 
economic-impact-of-ca-release-automation.pdf 
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No release automation tool will work in isolation. More advanced solutions 
will serve as an integration hub, orchestrating many activities across the pipe- 
line. At a simple level this could involve application-centric release automation 
to configure all the resources needed to support a new build (e.g., allocating 
server and storage capacity and ensuring an appropriate platform is in place 
to receive the build). 


Beyond addressing immediate operational requirements, advanced solutions 
will work in concert with many other processes to build a continuous deliv- 
ery ecosystem that helps IT achieve the most advanced levels of maturity. 
What distinguishes capabilities here isn't just strong integration, but the 
flexibility needed to support a more adaptive toolchain - one where new 
technologies can be quickly and easily incorporated to strengthen the con- 
tinuous delivery model. 


Tip To avoid vendor lock-in, ensure release automation tools provide an open and scalable 
platform, integrating easily with any continuous delivery toolchain for end-to-end visibility and 
orchestration of releases. 


Figure 6-2 and the section that follows illustrate and describe essential release 
automation toolchain integrations needed to optimize continuous delivery. 
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Figure 6-2. Release automation: toolchain integration 
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1. Requirements Design 


This integration allows agile teams to track multi-application release content 
through the software lifecycle and establish critical feedbacks loops for faster 
problem resolution and application delivery. 


With a real-time dashboard for managing and monitoring multi-application 
release content (user stories, features, and bug fixes) through the release pipe- 
line, agile teams gain complete visibility of release progress, more easily recon- 
cile dependencies, and can map to business requirements. 


Without this integration, agile teams would have to manually track and report 
on business-level user stories, features, or fixes to specific application releases 
moving through the pipeline. 


2. Service Virtualization 


This integration automates the launch of virtual services as part of a deploy- 
ment to optimize resources and speed testing. 


Here, DevOps practitioners can provision virtual services and execute test 
suites across multiple virtual environments directly within a deployment 
workflow. By deploying into any testing environment, teams are freed from 
constraints (e.g., waiting for physical hardware environments to be built and 


made ready for testing). This improves productivity and speeds time-to-value. 


Without this integration the release process could be interrupted. Manual 
requests would be needed to provision physical systems and virtualized ser- 
vices separate from the automated deployment workflow. This impedes the 
flow of value and ties up resources on repetitive and error-prone tasks. 


3. Test Data Management 


This integration automates the generation of accurate test data based on 
proper test cases within a release workflow. 


Without this integration, manual requests are needed to generate the proper 
test data separate from the automated deployment workflow. Again, this 
results in release interrupts, delays, and slower delivery. 


Note Integrating test data management with release automation should be considered a 
DevOps automation best practice. Not only does it ensure teams have ready access to accurate 
test, it also helps establish compliance (e.g., generating synthetic data to protect customer 
information) into the release process itself and avoid the delays associated with lengthy auditing 
checks at the end of each cycle. 
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4. Test Automation 


This integration automatically starts the test case process and ties the results 
back into the release to determine and confirm readiness for promotion. 


Here the test case process would be automatically initiated with the results 
linked back to the release. This is essential in order to determine go/no for 
automated promotion—enabling faster, higher quality deployments. 


Without this integration it would be necessary to manually determine if the 
application has sufficiently passed a testing stage in order to move forward to 
the next stage and then manually promote the application. Again, this is time 
consuming. 


5. Performance Monitoring 


This integration establishes monitoring earlier in the software lifecycle in 
order to feedback critical information needed to improve quality. 


Release automation can coordinate the installation and activation of moni- 
toring in pre-production. The technique of "shift left" monitoring (discussed 
in Chapter 7) enables teams to see the performance impact of releases and 
compare it against production baselines. This provides development with 
earlier warning on code-related performance issues and operations earlier 
guidance on service-level requirements. 


6. Existing Toolchain Investments 


A fully integrated continuous delivery toolchain solution will be open 
and scalable, coordinating the application of any existing products within 
standard and reusable release processes. Some important integrations 
include: 


e Continuous integration—Automatically kick off an applica- 
tion deployment upon the immediate completion of a 
software build in Jenkins. 


e Configuration management—Combine release automa- 
tion with solutions like Chef and Puppet to solve the 
problem of attempting a deployment when the target 
environment is not in a good known state. Integration 
here can be used to enforce specific environment con- 
figurations prior to deployment and manage configura- 
tion drift. 


e Cloud provisioning—Enable users to build workflows that 
provision, configure, and tear down cloud environments 
within a deployment workflow. 
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Release Automation: Capability Checklist 


With release automation playing such a central role in integrating tools and 
processes across the toolchain, solutions in this category should at a minimum 


deliver a deployment engine capable of supporting: 


To develop, plan, manage, and optimize the continuous delivery pipeline, 


Artifact management—The ability to deploy many differ- 
ent components and configurations of applications on 
physical, virtual, and public or private clouds. 


Configurable deployment options—A powerful, visual work- 
flow engine to easily create standard, reusable deploy- 
ment processes to promote apps from one environment 
to the next. 


Reusable deployment best practices—Shared components, 
allowing teams to leverage and reuse deployment logic 
across different projects and applications. 


Orchestration of preferred tools—As discussed, solutions 
should leverage existing tool and technology investments 
to automate deployments by using out-of-box action 
packs or through a software development kit. 


Deployment remediation and auditing—Using a visual 
dashboard teams can track and record configurations, 
artifacts, and release progress for improvement and 
auditing. 


release automation should also scale to helping teams: 


Design a shared pipeline—Orchestrate manual and auto- 
mated tasks within the continuous delivery pipeline. 


Execute many complex releases—Run through all the 
release phases—development to production for multi- 
app, multi-team releases. lterate and improve failed 
content. 


Plan and manage the timeline—Schedule and manage apps 
through multiple phases using a visual calendar. Provide 
immediate notification of conflicts and maintenance 
windows. 


Improve collaboration—Assign owners to tasks and use an 
activity feed to share comments. 
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e Manage and track content—Track features as they pro- 
ceed to production. Provide full insight when prioritizing 
and ensure the business implications of delays are clear. 


e Optimize releases—Detect problems in real-time, rec- 
ognize bottlenecks, and improve processes and team 
activities. 


As maturity increases, release automation should cater to more advanced 
requirements. This may include: 


Dependency Management 


When building multi-component/multi-application systems, there will be com- 
plex dependencies between applications or different versions of an application. 
This may include a mix of release, content, application, and application version 
level dependencies. The knowledge of which application version depends on 
which is critical and often only known to a small number of experts within the 
department. Systems should be able to establish the definition of these depen- 
dencies with automatic alerts when dependency conditions are not met. 


Pipeline Visibility with Notifications 


Systems should provide a clear view of the release pipeline, including all phases 
and all tasks within each phase. Each phase should show list of tasks, the order 
of their execution, and whether it is to be run sequentially or in parallel. To 
support continuous delivery, each release should trigger a new build and pro- 
mote this build through the pipeline, from the test phase and all the way to 
production. Automation should reiterate phases until all tasks pass predefined 
criteria. If something goes wrong with a release, delays and idle time should be 
reduced through automated notifications. 


Flexible Approval Processes 


For sensitive phases such as production deployment, full governance may be 
required. To support these cases, systems should prevent mistakes by allow- 
ing only the permitted users to approve the execution of these phases. 


Recommendations and Action Plan 


To attain the continuous delivery best practices described in this chapter, 
organizations need to ensure they apply due diligence when adopting an auto- 
mated approach. 
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As suggested, the best way to start is by assessing processes, culture, and tools 
currently in use. This way a clearer picture emerges of where businesses are 
today in comparison to where they need to be to support agreed goals and 
objectives. 


At the start of its journey, City Index used manual processes to deploy appli- 
cation code from development to production. Value chain analysis showed 
that moving code through development environments to quality assurance, 
then pre-production before finally going live, made up 50 percent of the deliv- 
ery effort? 


Tip When assessing capabilities, don't limit analysis to one element (e.g., test lab provisioning 
or configuration management). Take a system-level approach to understanding the flow of value 
and inhibitor across every stage—involving people, processes, and technology. 


Demonstrate Business Benefits and ROI 


Stakeholders, influencers, and decision makers need to understand the under- 
lying business benefits of adopting release automation tools to support con- 
tinuous delivery. 


Two key metric categories that are used to indicate IT performance and can 
be useful in supporting a case include the speed or throughput with which 
applications are delivered and the quality or stability of the releases. 


Tip Seek out real-world customer examples from companies that have achieved significant 
improvements in both release throughput and quality. ING is one such example. They increased 
release frequency to over 12,000 a month, achieving faster time-to-market with less than six 
weeks cycle time, but with a greater than 50 percent reduction in incidents.$ 


Any solution must also demonstrate positive economic impact to the depart- 
ment and business—both short term and long term. To support this, CA 
Technologies commissioned Forrester Consulting to conducta Total Economic 
Impact™ (TEI) study and examine the potential return on investment (ROI) 
that enterprises may realize by implementing CA Release Automation." 


*Full story: http://www.ca.com/content/dam/ca/us/files/case-studies/city- 
index-bets-on-ca-release-automation-for-it-operations.PDF 

http://www. slideshare.net/CAinc/case-study-ing-builds-highly-available- 
continuous-delivery-pipeline-with-microservices-and-containers 

"http: //www.ca.com/au/collateral/industry-analyst-report/the-total- 
economic-impact-of-ca-release-automation.html 
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To better understand the benefits, costs, and risks associated with an imple- 
mentation, Forrester interviewed five organizations that had implemented 
this solution in their enterprise. Taken as a whole, this composite company 
reported a 389 percent return on investment, $8.44 million net present 
value, with a 2.8-month payback. The study also illustrated that the com- 
posite organization's configuration management and testing team saved time 
and effort on deployments, with savings of six FTEs quantified at $1.22 mil- 
lion over three years. 


Note To help determine business benefits and ROI, seek out tools that calculate the full 
economic impact of release automation. Comprehensive tools provide total benefit analysis, 
assessing metrics such as increased staff productivity, reduced release errors, improved time-to- 
value, and reduced auditing and compliance costs. 


Execute Tactically, Grow Strategically 


Starting small is okay: It’s not in anyone's interests to embark on a lengthy 
company-wide committee to investigate introducing release automation. It’s 
often easier to showcase business value through a pilot. 


With this is mind, consider selecting a suitable project to act as your pilot. 
Many departments start small with a low-risk application that is important but 
not business critical. The aim is to start a groundswell of support, gather com- 
pelling metrics, and then apply lessons learned across larger teams and proj- 
ects. The Western Union Shared-Service Enterprise IT Operations team took 
a grass roots approach to DevOps adoption, starting small and measurable. 
The team used release automation tools to release software into production 
and then used this as a lever to open the door for broader conversations with 
its development partners.? 


Summary 


In this chapter, we discussed the automated methods needed to advance con- 
tinuous delivery maturity—taking teams from manual, scripted processes to 
more automated, standardized, efficient and agile methods, all while continu- 
ously improving both the quality of releases and the applications they deliver. 


*Presentation - https : //www. youtube. com/watch?v-JW2eukJuOqw 
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We also described how as the lynchpin in a continuous delivery ecosystem, 
release automation solutions must be capable of orchestrating many pro- 
cesses and tools. The ultimate goal is complete continuous delivery optimi- 
zation across the enterprise and zero-touch deployments, from planning to 
production. 


In the next chapter, we'll examine the DevOps strategies needed to build 
more agile operations— strategies that extend beyond basic monitoring of 
applications and infrastructure toward optimizing the all-important customer 
experience. 
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Agile Operations: Powering the Modern Software 
Factory 


In traditional enterprise IT, developers code and operations manage what 
comes “over the wall" to production. While DevOps regards this as the ulti- 
mate divisive anti-pattern, this practice has still been conducted for decades— 
but why? 


The model generally persists because of the nature of customer engagement. 
Applications have been generally designed “inside-out,” with customer inter- 
action through a single channel. Even if the channel is digitized, the focus is on 
improving business efficiencies, with customer benefits only considered as an 
afterthought. For IT operations, supporting this model has been difficult but 
manageable. 


But all this is radically changing. Now businesses understand that customer 
needs "outside" their organization must be brought "inside" and supported 
via omnichannel engagement. Omnichannel is all about continuity of experi- 
ence, regardless of where, when, and how a customer interacts with a business. 


From a commercial perspective, omnichannel provides an opportunity to 
enhance the all-important customer experience via new digital touch points; 
however, this increases IT operations complexity, with teams now faced with 
managing increased volumes of rapidly changing software services, delivered 
over modern and legacy applications and infrastructure. 


© CA 2016 
A. Ravichandran et al., DevOps for Digital Leaders, DOI 10.1007/978-1-4842-1842-6 7 
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With companies digitally “upshifting” from business efficiency to business 
model transformation, the value proposition of IT operations must change— 
from being good at managing the technology status quo, to becoming more 
agile and integral to driving successful business outcomes. 


This notion of an agile operation is highly synergistic with DevOps since it 
involves teams working collaboratively to establish a high-quality customer 
experience across the software lifecycle. Rather than wait until production 
and retrofit performance, an agile operations team works closely with devel- 
opment using new monitoring approaches to "bake" or craft quality into appli- 
cations—as they're engineered, tested, and released. 


Before examining new challenges and agile operations monitoring strategies, 
let's examine a case study where DevOps style practices have been used to 
great effect. 


Case Study: ANZ Bank 


Today, the only source of competition in the banking world is an obsession 
with serving and delighting customers. This is something Melbourne, Australia- 
based ANZ Bank had in mind recently when it launched a new application 
performance management (APM) program with the hopes of extending it 
across the entire IT infrastructure, which includes four data centers, main- 
frames, and more than 10,000 servers.' 


“The primary goal of all my teams is 100 percent availability of services for our 
customers. It’s simple as that," explains Adam Cartwright, head of IT Security 
and Operations at ANZ Bank. 


“That means whatever channel it happens to be—whether it's a corporation 
doing a payroll transaction or a private user transferring money using Internet 
banking—it's got to be up. It's got to be seamless. It's got to perform to their 
expectations. That is the primary mission of operations within technology" 


Unfortunately, the complexity of modern distributed applications means that 
this doesn't happen all the time, says Cartwright. Applications and systems go 
down, adversely impacting end users. In 2012, it became apparent that infra- 
structure monitoring alone—which focused on platform and event monitor- 
ing at the infrastructure level—was not enough to give ANZ Bank the insight 
it needed to fix and prevent incidents within the organization. 


"What you really need to do is to understand the transaction flow within the 
application context you're going to identify the root cause, or if you're going 
to get early signaling of potential problems before a customer actually has [an 
incident],” he explains. 


'Full story: http: //www.ca.com/us/rewrite/articles/management-cloud/customer- 
obsessed.html 
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For example, the bank has a distributed payment application that it used to 
transact billions of dollars for its highest value customers. The application 
comprises more than 120 distributed servers and 60 separate Java applica- 
tions. Whenever there was an issue with that application, it was nearly impos- 
sible to determine the root cause of the stability issues. The worst part was 
that many times ANZ Bank IT didn't realize there was a problem until its 
customers notified them about the issue. 


These challenges led to many of the goals of ANZ Bank's Application 
Performance Management (APM). First and foremost, IT wanted to reduce the 
number of incidents caused by application releases. This would increase the 
quality and confidence of application deployment or changes while improving 
the overall lifecycle of new ANZ applications. 


In addition, Cartwright wanted a way to perform deep dives into the applica- 
tion layers to obtain code-level visibility to measure performance and avail- 
ability. He also wanted to measure and analyze transactions as they moved 
across the distributed and highly diverse ANZ Bank infrastructure technolo- 
gies. These efforts have helped the bank minimize customer downtime. 


Using APM, IT is now proactively alerted when there's an issue before it 
affects service. Since APM allows the company not only to look at transac- 
tions flowing through an application, but also to identify the business user 
behavior attached to those transactions, outages can be found more quickly, 
and applications can be redesigned so outages don't repeat themselves. 


"Getting that insight in production is fantastic, but what is perhaps more bril- 
liant is getting that insight in the development-and-test environment,’ he says." Ve 
have been able to show that by putting APM into development and test—mak- 
ing it part of the process in those areas—you can stop defects from getting 
into production. And that's probably the most surprising thing to people new 
to APM, but the most critical thing from a production support point of view." 


Indeed, says Cartwright, this is one of the more astounding benefits of APM. 
For example, one project team was able to prevent 10 high-severity incidents 
from happening, saving more than a dozen hours of investigations. Another 
project team reduced recovery time from more than four hours to less than 
30 minutes with no impact to service. “While it's good to have APM and to 
use it to diagnose problems once they've occurred in production, you've got 
a customer that has been affected,” Cartwright explains. “It’s much better to 
stop that customer effect from occurring in the first place by not letting that 
sort of design issue propagate into production under transition." 


One of ANZ Bank's first projects in the test-and-development realm was a 
payments application that caused serious production issues. The bank had five 
or six major releases each year, and every time a new version went live there 
was “a raft" of high-severity incidents. “No matter what we did with testing, 
reviewing, traditional sorts of approaches, or performance and volume testing, 
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we always managed to end up in a situation where we had a release go live and 
we'd have problems in production," says Cartwright. 


Using APM, ANZ Bank eliminated between 10 and 15 high-severity incidents, 
which often stretched out three or four weeks following a release. “Now, 
we're down to one or two or, in some cases, zero [incidents];" says Cartwright. 
IT employees are thrilled with the change. 


"Since APM has been in, we have been able to pinpoint the exact impacted servers 
and restart these without impacting business and payments processing,’ explains 
Joseph Rocco, Support Transition Analyst at ANZ."'Before APM, the LMS [Limits 
Management System] recovery was four-plus hours. Post APM, it has been around 
30 minutes total to restart impacted servers with no outages.” 


This is an example of the fact that, while ANZ's main focus was on customer 
satisfaction, a positive side effect of reducing incidents is the boost it gave the 
IT organization as a whole."If you can stop having incidents, you track capac- 
ity and headcount within the organization to do other things, more proactive 
things," Rocco says." APM essentially means your organization is going to con- 
tinue to grow as your system footprint grows and the complexity grows." IT 
employees are freed up to be proactive rather than reactive. 


Following on from this success, ANZ extended its shift-left approach to more than 
20 applications. As a result, it has code-level visibility of performance and avail- 
ability issues, which not only stops defects from getting into production but also 
increases confidence in application development and deployment at the bank. 


Both development and operations teams undertake performance and load 
testing and correlate their data. As a result, pre-production efficiency has 
increased with a 60 percent reduction in time spent on solving software prob- 
lems, which equates to savings of AU $300,000.? 


More Change, More Complexity 


Progressive organizations like the ANZ Bank understand how customers 
expect rapid software iterations of new functionality together with high lev- 
els of performance. This fact was illustrated in an Enterprise Management 
Associates (EMA) report, which indicated that two-thirds of organizations 
who have embraced continuous delivery release code weekly or even more 
frequently.’ But this is not without its problems, with the EMA report also sug- 
gesting that development now spends as much time supporting production as 
it spends writing new code, while operations spends more time on application 
support than on any other single task. 


?https://www.brighttalk.com/webcast/7819/134027 

“Omnichannel, Microservices, and Modern Apps,” January 2016: http: //www.ca.com/ 
content/dam/ca/us/files/white-paper/ema-omnichannel-microservices-and- 
modern-applications.pdf 
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This support now extends to managing modern microservice style architec- 
tures. Designed to be deployed as discrete elements (or services) performing 
a specific set of tasks and running as its own process, microservices break 
down specific functions into small components connected via APls. 


While this approach potentially allows services to be updated more regularly 
without impacting other elements supporting a business process, there are 
major operational challenges. Not the least: 


e Increased diversity—VWVith microservices, developers can 
code in multiple languages and work with databases best 
suited for their service. For operations, this means main- 
taining performance and availability over unfamiliar tech- 
nologies like Node.js and MongoDB. 


Tip Always remember that with modern digital systems applications supported by 
microservices and newer technologies, application performance management solutions must be 
more resilient and scalable than whatever they are monitoring! 


e Massive complexity—One monolithic application can 
become thousands of microservices. Unlike monoliths, 
microservices make visualizing application topologies and 
transactional flow using traditional tools extremely chal- 
lenging. Add the prospect of potentially running multiple 
service versions in parallel and monitoring complexity 
increases exponentially. 


Much of the microservice complexity lies in the rela- 
tionship and APl-centric communication between 
services. With distributed systems like these, teams 
must consider a whole range of new issues, including 
network latency, asynchronous messaging, and load 
balancing, not to mention end-to-end performance 
issues when microservices connect with back-end 
applications. 


e Increased noise—With mobile apps, microservices, and 
containerized environments, the volume of alarms and 
alerts can grow significantly. Trying to find filter out noise 
and find the root cause of problems using traditional 
rules-based approaches becomes much more difficult. 
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Tip Consider periodically assigning developers to review the alerts and log messages their 
code is producing. This could be a useful way of identifying where refactoring work or improving 
application supportability is required. 


e Ephemeral nature—With the pace of change necessitat- 
ing far shorter application lifespans, triage teams no lon- 
ger have the luxury of capturing and analyzing historical 
data using a plethora of tools. Teams require methods 
to better understand real-time performance across mod- 
ern architectures (including containers), which in more 
dynamic environments might be changing in a matter of 
minutes, even seconds. 


New IT Operations Imperatives 


IT operations as a discipline will no longer be judged on how effective it is 
at fixing application problems, but rather on the ability to improve business 
outcomes—detecting and fixing issues, yes, but working collaboratively with 
other teams across the software factory to establish quality. 


What's admirable in the case of ANZ is how managing to business outcomes 
has become an established part of the IT operations mantra. True, the team 
is still responsible for maintaining stability and resilience, but by establishing 
performance monitoring in areas beyond their traditional control, quality and 
confidence have increased substantially. 


Rather than reactive break-fix approaches to monitoring, a DevOps focused 
operations function will leverage advanced tools to proactively ensure a qual- 
ity customer experience before a system reaches production. In this sense, 
practitioners will become uber sysadmin craftsmen and as agile as their 
development colleagues. And with developers increasingly empowered to 
make operational decisions, a move toward the agile operations approach will 
become more important. 


So what new skills will agile operations teams need to acquire? There are a 
few important ones to consider, covered in the following sections. 


Proactive Engagement 


The IT operations landscape changing is dramatically with many advances 
in technology, but not necessarily changes in values or thinking. With the 
democratization of operational functions, a good agile operations focused 
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practitioner will be one that strives to intimately understand the behavior 
of applications, nurture production systems, and feedback information and 
knowledge. To this end, agile operations craftsmanship will be less about pull- 
ing out router cables and console watching and more about analyzing app 
behavior and the customer experience to drive improvements. 


Designing for Failure 


The traditional approach of measuring operational effectiveness in terms of 
preventing failure doesn't work anymore. With cloud applications, there are 
many moving parts, in terms of technology and process. There will be mobile 
apps and APls supporting new digital channels, but also back-end data and 
systems. In these environments, failures are inevitable, so the objective should 
be to design for them— containing problems, but still keeping the business 
running. 


Accepting this reality, agile operations will work closely with development to 
mature the software engineering and monitoring practices needed to opti- 
mize modern cloud-based systems— such as, for example, establishing infra- 
structure monitoring with every release or exposing performance diagnostics 
with every application build. 


Tip Make it a rule rather than the exception to establish infrastructure and performance 
monitoring in every environment—right from the moment something is provisioned or placed in 
maintenance mode. 


Moving Beyond Resilience 


Like the mythical Phoenix, modern cloud systems and microservices should 
be designed to bounce back from every situation. And they must, because 
when a business uses these approaches to engage customers at scale and deal 
with unknown demand, there'll be much more complexity—at the very least 
ensuring hundreds (perhaps thousands) of services continue to be available 
and performant. 


Throw in complex architectural issues like asynchronous messaging and API 
latency and there's a whole new world of pain. It's analogous to cutting a head 
off the mythical Hydra. Just when you've addressed one problem, two more 
grow in its place. This is why resilience doesn't go far enough. 


When considering new digital systems-of-engagement, applications can't 
only be resilient, they too have to be Hydra-like. Rather than fixing problems, 
agile operations methods work to improve applications both technically 
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and commercially after every significant event (positive or negative). This is 
especially important for mobile apps and why experience-based analytics is 
becoming increasingly important. 


Making Support a Top Design Issue 


Rather than working as downstream production traffic cops, agile operations 
will engage upstream with developers in an advisory capacity. Of course this 
requires development to be fully involved; having the desire to learn from IT 
operations in order to design and build systems that are much easier to sup- 
port once in production. 


But taking the time to learn is challenging if operations provides no useful 
information or the tools used only address monitoring from one perspective. 
I'll be difficult too, if in the rush to meet project level goals, development 
teams select their own point tools and methods at the expense of overall 
system-level resilience and performance. 


Combatting these issues requires ending the “toe-to-toe” battles with devel- 
opment at various checkpoints across the software lifecycle (which never end 
well). It involves demonstrating how the knowledge and expertise everyone 
has acquired over many years not only helps improve application supportabil- 
ity, but also makes peoples jobs easier and more rewarding. Some examples 
include: 


e İn a mobile app scenario, presenting live usage crash ana- 
lytics that developers can use to identify where functional 
improvements may be needed. 


e Sharing APM toolsets that enable developers to review 
issues from their perspective. Serving up information in 
their terms, about their world, with their code. 


Tip Consider organizing joint dev and ops workshops where teams openly discuss and share 
the "cool" things they've learned over many years. This includes operations sharing information 
about improving resilience and development explaining the methods used to release software 
updates in small batches. 


e Openly discussing how older style alert and static base- 
lining make less sense in monitoring modern dynamic 
environments and only increases the support burden. 
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e Jointly conduct triage scenarios, simulating what's really 
involved when attempting to fix application issues at 
3:00am (clinically review tools to determine whether any 
of the "noise" actually warrants getting support staff out 
of bed in the small hours)! 


Tip Use workshops and tools to: 1) help developers understand the “on-call” support 
implications of their designs, and 2) help operations understand what information developers need 
to make performance improvements. 


Active Monitoring 


Teams need to embrace active monitoring methods to build an understanding 
about issues before they affect customers. Part of this involves finding better 
ways to remove misleading alarms and false-positives. 


Traditionally, monitoring solutions have dealt with false-positive alerts using 
performance baselines. Although this has helped, these approaches typically 
look at only one part of the issue: severity. A different way to look at the issue 
is to analyze both severity and duration. For example, a minor issue occurring 
over a long period of time could eventually escalate into a larger issue that 
teams need to investigate. Alternatively, a medium issue that occurs some of 
the time should raise an alarm because it could become a larger problem very 
quickly. 


Tip Once techniques to remove irrelevant noise and alerts have been applied, look for ways 
to combine active alerting with methods to capture more detailed information. For example, 
automatically initiating a diagnostic transaction trace. 


Toward Agile Operations 


The 2015 Freeform Dynamics report, produced in association with CA 
Technologies, indicated that with DevOps, 63 percent and 61 percent of 
advanced adopters, respectively, were better able to help the business act 
swiftly on digital opportunities and attack and defend more quickly—with 77 
percent and 72 percent also indicating improvements in achieving customer 
acquisition and retention goals.* 


^https://www.ca.com/us/rewrite/articles/devops/assembling-the-devops- 
jigsaw.register.html 
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This suggests organizations can move fast without sacrificing high quality. 


Achieving this requires IT operations adopting many practices and processes 
familiar to their agile development colleagues. Of course, this still requires 
monitoring the performance of production applications, but it also means 
ensuring information gained is fed back and incorporated into agile develop- 
ment processes (e.g., agile sprints). See Figure 7- |. But, and as we've described, 
IT operations must ensure that the information being shared with develop- 
ment is of higher value than that provided through traditional system alerting. 
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Figure 7-1. Agile development and operations 


From a development perspective there are many new tools and techniques 
that support shifting-left phases so teams can work in parallel to meet the 
goals of continuous delivery. For example, and as described in Chapter 5, 
by simulating dependent systems, service virtualization solutions help teams 
remove constraints, thereby providing teams immediate and realistic test 
environments. 


IT operations has to a lesser extent been slower adopting this approach. This 
is often due to keeping watch over a myriad of diagnostic tools—meaning less 
time is spent delivering information that's actually useful to their colleagues. 


But failing to embrace an agile operations shift-left approaches can lead to 
cost increases and missed business opportunities. For example, hurriedly pur- 
chasing additional capacity due to unexpected performance problems, or hir- 
ing more contractors because of increased staff burnout and turnover after 
excessive after hours support. 
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Shift-Left Monitoring 


Adopting agile operations methods like shift-left can be a daunting prospect 
when people feel ill-equipped to cope with any type of change. According to 
an Enterprise Management Associates (EMA) report, fewer than 50 percent of 
IT professionals are confident that their application management solutions can 
adequately meet the monitoring requirements of modern IT environments.’ 


It's important therefore to resist the urge to add another tool (be that commer- 
cial or open source) to a growing arsenal of monitoring tools. This approach 
can further increase team fragmentation, as confirmed in an Infrastructure & 
Operation Trends Survey, where 80 percent of respondents agreed that dis- 
jointed, cross-platform management leads to lost opportunities. 


As the survey alludes, this can be suboptimal from a business perspective, 
especially when incomplete (or incomprehensible) performance signals result 
in bad decisions. For example, snap purchasing servers due to an unexpected 
performance condition. This might have addressed the immediate issue, but 
the organization has just reduced its profit margins and the real problem still 
lurks-somewhere. 


This suggests that monitoring approaches should be conducted from an 
"avoidance" perspective. That is, avoid problems by detecting them early and 
often, before they can impact the business. Do this with fewer tools or spe- 
cialists and cost encumbrances are avoided too. Achieve and demonstrate 
repeatedly, and IT operations becomes seen as less of cost-center and more 
as a value-generator, adding to the bottom line. 


Continuous High-Quality Feedback 


Enabling a continuous cycle of feedback is fundamental to an agile operations 
shift-left approach and the success of a DevOps program. In its simplest form, 
feedback from operations should help development toward reducing the vol- 
ume of code defects, while feedback from development should guide opera- 
tional service-level requirements before an application goes into production. 


That's easy to say but difficult to achieve in practice. Applications are more 
diverse, distributed, and ephemeral, meaning feedback must be faster, richer, 
useable, and useful. To support this, modern approaches must clinically remove 
noise, distill intelligence, and then (and this is the really important part) put it 
in context of the people having the most to benefit from it. This way feedback 
is more valuable because it's actionable. 


*"Omnichannel, Microservices, and Modern Apps,” January 2016: http://www. ca .com/ 
content/dam/ca/us/files/white-paper/ema-omnichannel-microservices-and- 
modern-applications.pdf 

sSettling the Settling the Breadth vs. Depth: http: //www.ca.com/content/dam/ca/us/ 
files/ebook/settling-the-breadth-vs-depth-debate-mfinfrastructure. pdf 
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Take for example a mobile shopping app update with new social network API 
that's going to be released to take advantage of Black Friday. The new API is 
potentially a great way to increase revenue, but with demand difficult to pre- 
dict, how do developers know that their code will handle back-end latency 
and load? We could take the "suck it and see" approach or stack the data 
center with more capacity, but with brand reputation on the line, is that a risk 
worth taking? 


Alternatively, we could use agile operations thinking—serving up developers' 
critical insights into API latency, back-end load issues, end-to-end transac- 
tion times, mobile device performance impacts, every time they commit their 
code. This way, feedback is provided exactly when they have the most to gain 
from it—as they develop. And since it's placed in context of the business goals 
they're looking to support, it's immediately actionable. This enables the entire 
team to act with more purpose and urgency. 


Scenarios like this play out all the time in the real world, with agile operations 
approaches providing the cross-functional glue needed to build a shared under- 
standing of both problems and opportunities for improvement. For example, 
by using monitoring tools in performance benchmarking, teams can better 
address capacity and scalability issues; gaining confidence, new applications 
under development can cope with production load. This was a significant ben- 
efit that Danish Retailer Dansk Supermarked realized when implementing an 
APM solution to monitor development, test, and production environments." 


By facilitating the sharing of performance information, agile operations help 
teams move beyond making small incremental improvements in operational 
efficiency toward becoming a business differentiator. This is evidenced by 
a Techvalidate report, which indicated that more than half of organizations 
using APM stated it has helped them proactively manage user experience to 
create competitive advantage.? 


Intelligence and Analytics 


As operational functions become fused with development, advanced analytics 
and statistical methods will play a key role in serving DevOps practitioners 
the essential information needed to drive improvements. 


Rather than attempting to process a mountain of alerts to gain clarity over the 
current state of performance, cross-functional teams will newer techniques to 
better predict application performance and usage. Armed with these insights, 
teams will make smarter and faster decisions, not just in a production context, 
but across the software lifecycle (see Figure 7-2). 


"http: //www.ca.com/content/dam/ca/us/files/case-studies/dansk-supermarked- 
group-safeguards-online-sales-with-ca-apm.pdf 
*https://www.techvalidate.com/product-research/ca-application- 
performance-management 
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Some valuable methods include: 


e Mobile app analytics crash reports—Help ensure any reli- 
ability issues impacting user experience are quickly 
resolved. With device and platform specific analytics, 
developers can optimize code for uniform performance 
across all mobile endpoints. 


e Behavioral analytics—By understanding how mobile apps 
perform under real-world conditions (e.g., slow wireless 
networks), developers can optimize the user experience 
(UX) designs. 


e Usage and performance analytics—These enable develop- 
ers to improve design with every new iteration, ensuring 
what they build stays performant as usage and conditions 
change. 


e Business analytics—These can increase the chances of an 
app meeting business targets when it moves into produc- 
tion. Analytics can also be used to measure ROI from 
different applications to help prioritize future develop- 
ment (e.g., revealing how newly developed features and 
functions are helping increase customer engagement and 
preventing churn). 


DEVELOPMENT | —  ITOPERATIONS 
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Figure 7-2. Agile operations analytics—quality and resilience across the software factory 


With the digital-business landscape shifting continuously, predictive analyt- 
ics will be supplemented with prescriptive techniques that can actually guide 
practitioners toward finding the best course of action in any given situation. 
And with less and less time to conduct protracted root-cause exercises, tech- 
niques like these will become essential for agile operations. 


— 8] 


Chapter 7 | Manage 


In an APM context, it's easy to see immediate usefulness for complex tri- 
age—prescribing solutions as problems emerge. But as these technologies 
mature, they'll also become more business-centric. Consider for example hav- 
ing the ability to predict a missed online sales target due to a slow developing 
performance problem and then quickly prescribing necessary changes in both 
application (design, function, etc.) and infrastructure to prevent the problem 
from occurring. 


Providing business outcome-based capabilities like these will become the ulti- 
mate litmus test of successful agile operations and a shift-left strategy. 


Agile Operations Tooling 


To deliver agile operations and shift-left approaches we've described, organi- 
zations need to ensure they have the right application performance manage- 
ment strategies and tooling. Before looking at the tools themselves, and as 
illustrated in Figure 7-3, let's outline some major considerations: 


Eartier Warning tor Dev on operational issues before system S Ed Earlier Guidance for Ops on production performance 
' goes live requirements & KPI's am 
' i 


Business 
Conversion 


Customer End-to-end Transactions 


Engagement 


' 
3 Quality enhanced at | Seniceprintized according to business impact/customer ' 
critical "moments of truth" experience d» 


Consolidate tools to @ Gain critical insights uture proof the business 
remove the noise Sp) with analytics at] through scale and extensibility 


Figure 7-3. Agile operations and shift-left monitoring—essential strategies and tools 
capabilities 


Early Warning for Business and Development 


Agile operations tools should provide key insights into the performance of 
business transactions before the system goes live. With business traversing 
APIs, mobile apps, web infrastructure, and back-end systems-of-record, com- 
prehensive views into performance across the entire application and infra- 
structure fabric become essential. This enables development teams to quickly 
understand infrastructure dependencies, how functional changes impact busi- 
ness performance, and where refactoring is needed. 
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By way of example, let's consider how integrating APM with continuous inte- 
gration tools (e.g., Jenkins) can help developers. By providing APM data dur- 
ing a build process, developers can quickly determine the impact their code 
will make on performance (the early warning) and then leverage it to enact 
improvements. For example: 


e Immediately see API usage increases between builds. 
Payback: Avoid increased costs in API pay-per-use 
scenarios. 


e  |dentify authentication service overuse in new mobile app 
development. Payback: Improve customer experience. 


e See how upstream functional changes cause downstream 
performance issues. Payback: Determine risks associated 
with the build. 


The APM data provided should also be extremely granular (e.g., methods 
and transactions) and allow developers to roll back to see the impact of any 
code-level or environmental changes on performance. Additionally, by incor- 
porating APM with continuous integration pass-fail build tests, high quality is 
established early (see Figure 7-4). 


Note Using bi-directional integration, this process also updates APM, allowing operations to 
establish a complete performance "system of record" before production. 


Check in code more frequently -with APM data, Jenkins runs the build and tests it more thoroughly in 
pre-production to ensure high -quality releases 
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fail based on APM 
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Figure 7-4. APM and Jenkins integrated workflow 


In a mobile app context, agile operations tools should incorporate capabilities 
that help business and development teams better understand and respond to 
customer behaviors during mobile engagement. Here, end-to-end transaction 
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performance monitoring should be supplemented with analytics that reveal 
how new functions are driving increased engagement levels, activity, and cus- 
tomer retention—data that is critical when aligning development efforts to 
business initiatives. 


Early Guidance on Operational Impact 


This involves employing tools that continuously track business transactions 
and customer engagement to guide operations on production system require- 
ments and KPls. This requires discussion and shared agreement on what's 
important and why, collaboratively developing a shared set of metrics against 
which the system as a whole (comprising application and infrastructure com- 
ponents) can be monitored. 


Difficulties may arise when architectural complexity and new emergent appli- 
cation behaviors make it difficult to establish accurate baselines against which 
application performance can be measured. Too often, best-guess performance 
baselining results in intermittent but acceptable performance spikes (caused 
by new application functionality) flooding monitoring systems with alerts. In 
other cases, longer-lived problems indicative of serious code defects go unno- 
ticed because they fall within tolerance levels. To address this, teams should 
consider mechanisms that rely less on subjective or best-guess baselining, 
preferring instead statistical methods such as differential analysis where pat- 
terns of performance are built and monitored using multiple weighted crite- 
ria. Then, when thresholds are breached, deeper analysis (such as transaction 
tracing) can be initiated to pinpoint the problem and provide immediate and 
accurate feedback. 


Prioritize by Business Impact and Customer Experience 


Today's applications rarely function in isolation. In an online flight booking sys- 
tem, the overall customer experience can be delivered by 10 or more discrete 
elements delivered across multiple channels—from selecting a seat online and 
checking luggage at a kiosk, to making a payment and scanning a boarding pass 
with a mobile app. 


The elements supporting this "experience" could be transactional, contex- 
tual, or a mixture of both. Applications and infrastructure will be equally rich, 
including APIs, mobile apps, wireless networks, sensors, and kiosks, along with 
legacy booking and CRM applications. It's essential therefore that agile opera- 
tions tools deliver deep diagnostics across all these technologies, plus the 
ability to unify and monitor information at a higher service level.In the case of 
a flight booking system, for example, being able to prioritize a seemingly unim- 
portant API latency problem because of its significance in supporting a new 
revenue generating service (e.g. a car rental booking service from a partner). 
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Feedback at Key Moments of Truth 


In all environments, agile operations tools should provide teams fast feedback 
on software code effectiveness and problem components. This feedback is most 
critical at key points impacting customer experience (moments of truth). By 
leveraging live app experience analytics, for example, operations can provide 
invaluable information to developers and business analysts about the effective- 
ness of new code in improving user engagement, activity, and retention. 


In cloud environments where application resources are invoked according to 


demand, cost, time, or other business metrics, feedback may be more difficult. 


Often, when performance problems arise the first question asked is “what 
changed,” which can lead to finger pointing and conflict. While many tools can 
detect changes, they still require lengthy analysis to determine if the change 
caused a problem. More modern solutions address this by placing changes in 
context of application performance, allowing teams to roll back to a point in 
time to determine what changed and the knock-on effect on performance. 


To support these strategies, modern monitoring solutions should deliver the 
following capabilities: 


e Noise removal—Distill and simplify complex application 
topologies into role-based views based on elements such 
as application component, location, and business unit. 
Using these and other attributes, developers and support 
analysts can quickly reorient toward addressing issues 
and enacting improvements in context of the task at hand. 


e Analytical insights—As discussed, provide all stakehold- 
ers actionable information across the software lifecycle. 
With app experience analytics, for example, advanced 
geo-spatial services allows business analysts to build per- 
formance and usage patterns, while developers can use 
video app playback to better determine the impact of 
newly introduced functionality on customer experience. 


e Uninterrupted visibility—Deliver end-to-end transaction 
level visibility from the mobile app to the mainframe. This 
visibility should be provided in from a customer experi- 
ence perspective with full and immediate traceability into 
problem areas across the supporting applications and 
infrastructure (e.g., network response time and API calls). 


e Scale and extensibility—The only way to effectively engage 
customers and transact more business at scale is through 
mobile and cloud. Monitoring therefore needs to be 
equally scalable—future-proofing business by seamlessly 
supporting new technologies as they're introduced. 
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e Manage to outcomes—Strong consideration should be 
given to advanced tools that better determine what cus- 
tomer experience improvements (business outcomes) 
can be achieved by aggregating technology diagnostics 
and optimizing performance (outputs). This is key to 
ensuring the actions of cross-functional teams are fully 
aligned toward driving business improvement—not man- 
aging technology for technology's sake. 


Summary 


The days of IT operations working in the closed confines of a network opera- 
tions center and just keeping the technology "lights" on are numbered. 


As software releases grow in volume, variety, and velocity, agile operations will 
emerge as a key DevOps enabler. It’s a collaborative discipline where practi- 
tioners work across the software factory to help business achieve the best 
outcomes from a high-quality customer experience. 


Practitioners of agile operations will become true DevOps craftsmen; employ- 
ing new methods, modern tools, and advanced analytics to deliver superior 
application performance without increasing the support burden. 


In the next chapter, we'll discuss how DevOps can coexist with existing meth- 
odologies and best practices. We'll also examine its impact on other IT func- 
tions, including enterprise architecture and security. 


III 
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Practical DevOps 


Leveraging Existing Reference Sources, Roles, 
and Practices 


In enterprise computing, DevOps will never operate in a vacuum. Over many 
years businesses have invested, adopted, and adapted many other methodolo- 
gies and practices. For DevOps to be successful, this means many practices 
and existing roles (beyond development and operations) should be care- 
fully reviewed and, if necessary, adjusted to drive improvements across the 
DevOps-enabled software factory. 


DevOps and Enterprise Architecture 


For many years, organizations have understood the dangers of technical debt. 
That is, the additional overhead arising when badly designed, poorly tested, 
and defect-ridden software is accepted for short-term gain. Analogous to 
financial debt, too much technical debt and the associated interest can cripple 
an organization to a point where they're constantly putting right previous 
wrongs at the expense of delivering new innovations. 


Architectural debt is similar to technical debt and equally problematic. Poor 
architectural decisions can severely limit an organization's ability to move 
toward more agile styles of delivery. The result can be lower levels of innova- 
tion, slower time to market, or more cost and effort consumed rebuilding appli- 
cations. These poor decisions are the consequence of conflicting architectural 


© CA 2016 
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perspectives. On the one hand, scant regard for architecture and unconstrained 
development leads to software that is hard to integrate, support, and enhance. 
On the other, overly defined architecture is difficult and complex to implement, 
leading to delays in software delivery. 


Some DevOps practitioners feel that they don’t need to be guided Enterprise 
Architecture (EA). Conversely, many Enterprise Architects believe that rigorous 
dictates, methodologies, and practices must be adopted fully by DevOps pro- 
grams. In truth, both agile development and DevOps cannot succeed without 
EA. However, the scope and application of the discipline must change to acceler- 
ate DevOps benefits without burdening the business with additional risk. 


Without Good Architecture IT Builds Software 
Slums 


The UK high-rise tower blocks built circa 1950-1980 were heralded as archi- 
tectural wonders. Large towers housing the same population as the smaller 
pre-war houses they replaced. With large rooms, excellent views, and sur- 
rounding open spaces, they were initially lauded as the low-cost future for 
urban housing. However, there were many problems. 


Due to cost cutting, substandard materials, demanding deadlines, and rushed 
construction practices, these modern-age housing miracles quickly became the 
new slums of the day. And by building tower-blocks across the country, town 
planners unwittingly replicated bad designs everywhere. The result: undesir- 
able housing, elevated crime levels, and urban decay. Even the surrounding 
open areas and playgrounds were neglected because no one had ownership 
over maintenance or supervised them. 


In many ways, this is analogous to many architectural design calamities made 
by IT in the past. 


Built over many decades, rigid monolithic application designs have become 
difficult to scale and require costly maintenance. They're also tough to police 
and secure, with vandalism and theft being a constant problem. Add to this 
integration issues (IT's own unsupervised open spaces) and we're left with sys- 
tems that are inadequate to meet the needs of modern digital business. That's 
not to say, however, that modern design approaches are the panacea. Web and 
microservices introduce new architectural issues, not the least, dependency 
management and monitoring complexity. 


Enterprise Architecture Must Adapt to the Times 


DevOps, with its focus on collaboration across the entire service lifecycle, is 
now seen as the answer to many of these issues. This is fine in principle, but 
without flexible EA, the end result could be IT developing application “slums,” 
only more of them at an increased pace. 
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With some justification, many DevOps practitioners argue that heavy EA 
practices and frameworks haven't evolved to match the pace of agile. Iterative 
style development essentially means an end product very different from the 
initial idea, which necessitates a transition from set-in-stone architectures to 
a more fluid mix based on continuously evolving decisions. And if architects 
can't adapt, autonomous agile teams with easier access to open source and 
cloud-based resources are increasingly empowered to make architectural 
calls. But this can be problematic. 


Great developers don't necessarily make great architects; they're not hard- 
wired that way. Charging down a development path without architectural 
guiderails, or tactically making team-based decisions, could address short- 
term development requirements but compromise broader program-level 
objectives. For example: 


e Scalability and performance issues needed to address 
the business strategy of tripling the customer base and 
increasing satisfaction scores are neglected because 
development only concentrates on delivering more 
functionality. 


e Developers acquire public cloud resources to accelerate 
application testing, but don't consider masking personally 
identifiable customer details when moving test data. As a 
result, organizations (especially those in heavily regulated 
industries) risk compliance breaches and heavy financial 
penalties. 


e Because of team-based technical bias, a development 
group chooses one NoSQL database over the technology 
already used by another team. The existing technology 
would have met their needs (albeit with compromises), 
but their technology bias has significantly increased the 
support burden on IT operations. 


It’s important not to lay the blame squarely on development. Architects should 
recognize that any misalignment between existing processes (particularly the 
rigid ones) and the vision of an increasingly software-driven business will only 
further isolate the practice to the eventual detriment of the business. 


New Fluid Guidelines and Principles 


By incorporating the experience of enterprises architects within DevOps 
teams, organizations can maintain the pace of software delivery without intro- 
ducing chaos. What's key, however, is to limit architectural over-engineering 
and provide development groups a minimum set of EA so as to avoid technical 
and architectural debt. 
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Even in minimal-mode, EA can still help developers quickly identify critical 
software design issues. The aim being to guide developers into understanding 
what architectural facets make up a successful application, and more impor- 
tantly, how new ways of thinking can support more sustainable software 
innovation. 


Interestingly, these practices are analogous to those that could be used to 
avoid building “housing slums”: 


Avoid substandard or unsupported materials—Development tools will 
most likely comprise both commercial and open source software and utilize 
both on-premise and cloud-based infrastructure. Teams must ensure applica- 
tion supportability becomes a key consideration, especially as more modern 
applications comprising multiple components move into production. 


Note Just as the cost of maintaining UK housing tower-blocks reached unsustainable levels, 
so too will support burden on the organization when open source software isn't maintained and 
enhanced by the broader software community or lacks commercial backing. Enterprise architects 
should also work closely with development to understand where constraints lead to testing 
compromises—the "construction shortcuts" syndrome, resulting in compliance and quality issues. 


New technologies shouldn't rehouse old problems— Enterprise archi- 
tects should avoid rigid and inflexible edicts on technology usage. For example, 
mandating that all applications must be containerized, even legacy applications. 
This could be extremely problematic since the runtime-independent nature of 
containerized applications will extend the life of “problems” and provides no 
incentive to clear existing technical debt. 


Note As UK high-rise housing tower-blocks degraded and vandalism increased, many local 
authorities tried to contain the situation by housing "problem groups" in the same units. Enterprise 
architects should avoid the same trap as they adopt technologies such as public cloud services 
and containers. 


Monitor and manage sub-contractors— Whatever cloud model is 
adopted, abstracting away the infrastructure or application stack frees up 
development to focus on coding. But this shouldn't mean relinquishing vis- 
ibility and control over application performance and the end user experience. 
To this end, enterprise architects must act as cloud-brokerage advisors. For 
example, working to ensure that cloud service providers incorporate open 
APIs with their offerings so that monitoring can be seamlessly incorporated 
into existing tools. 


DevOps for Digital Leaders 


Building new digital services with DevOps is only part of the role of EA. The 
innovations of today may become legacies of the future, while many new apps 
must also integrate with existing systems that have strict compliance and risk 
controls. It's essential therefore that architecture covers both bases, working 
in minimal mode to ensure fast construction of quality services, but also apply- 
ing standards and governance when needs dictate and systems change. 


Actions to Establish EA in DevOps Programs 


Some important steps needed to ensure EA becomes a sustainable contribu- 
tor to a DevOps initiative include: 


e Communication—To impress the importance of flexible 
architecture practices and standards. For example, API 
instrumentation shouldn't be mandated with rigid rules, 
but by carefully outlining how the practice improves soft- 
ware quality and supportability. This involves building 
closer ties with developers to ensure the right tooling 
decisions are being made. Enterprise architects should 
always emphasize that their participation isn't to slow 
down one particular team, but to ensure that team deci- 
sions (especially tooling) support broader program goals 
and objectives. 


e  Collaboration—ln dynamic agile environments, it's natu- 
ral for small teams to only focus on their own project 
and not consider the wider business context. Enterprise 
architects must apply flexible governance to ensure all 
stakeholders are involved in decision making without 
the system becoming overly bureaucratic. To this end, 
the governance approach must outline the need for big- 
picture strategy at an overall application portfolio level 
(complete with funding, commitment and risk manage- 
ment), together with support at a project level so to 
drive better outcomes in a business context. This sup- 
port should start at the requirements phase and extend 
across the software development lifecycle. 


DevOps and Information Security 


It's a common misnomer that DevOps only involves closer collaboration 
between development and IT operations teams. In actuality, DevOps pro- 
grams must also involve other disciplines that have traditionally been engaged 
late in the software development lifecycle. This is especially important with 
regard to information security. 
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At first glance it appears that the goals of DevOps and security are at odds. 
Whereas DevOps calls for increasing the delivery of high-quality software, 
security and compliance seeks careful and deliberate oversight to ensure the 
business isn't opening itself up to vulnerabilities. And with a mountain of rules 
and regulations to support, it's not surprising that security could easily become 
being regarded as another bottleneck in release and deployment processes. 


All teams must accept that security is a key facet of "high-quality" software, 
which again can be established without slowing down development. There are 
four essential practices to consider: 


Make everyone accountable for security—DevOps impresses the need 
shared responsibility and accountability. Therefore, security professionals 
should seek to build relationships with dev and ops teams and engage them 
as active stakeholders and participants in driving security improvements. As 
with enterprise architecture, this doesn't mean continually enforcing rigid 
and inflexible policies, but actually working collaboratively to assign security 
responsibilities to the team's best positioned to act on them. For example, 
during every application security incident, developers responsible for the 
actual code implicated should really be the first group called to help address 
the problem. These teams will be much more familiar with the software work- 
ings, plus the lessons they learn will help harden application security. 


Demonstrate how DevOps improves security and vice versa—As 
organizations increasingly embrace DevOps, there'll be many new automated 
tools and practices introduced. As with everything new, these elements could 
introduce new threats and risk. Rather than see this as a problem, highly col- 
laborative teams should work proactively to identify where additional guid- 
ance and controls are needed and can be applied without causing friction. 


Take the development of a new mobile application for example. Here security 
experts can provide critical guidance on new threat surfaces, API governance 
requirements, and vulnerability testing. It’s also important to consider that 
many new tools introduced (especially in areas like configuration manage- 
ment and release automation) also provide an opportunity for teams to build 
and improve security within the continuous delivery pipeline. To this end, 
it becomes less about making DevOps more secure and more about using 
DevOps (and especially automation) to improve security. This could include: 


e Invoking techniques such as static code analysis during 
every application build, or providing development teams 
with comprehensive and fully automated security testing 
services that can be used repeatedly. 


e Automatically creating the minimum set test cases with 
maximum security test coverage, right from the earliest 
stages of software development: the requirements phase. 
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e Reducing security test cycle preparation time by request- 
ing and reserving accurate and compliant data from a test 
data repository. 


e Generating realistic synthetic test data and incorpo- 
rating directly into virtual or emulated services so as 
to improve testing quality while avoiding compliance 
exposures. 


Shift security ‘“left”—As with the traditional development to opera- 
tions code handballing, the tendency has been to engage security very 
late in the development process. Too often, security teams are seen as 
the bottleneck police, holding up deployment with snap code audits and 
lengthy compliance checks. DevOps practices, however, enable security to 
be established during parallel development and testing. As code is devel- 
oped, automated tests can be automatically invoked to continuously check 
and demonstrate compliance controls. This could include separation-of- 
duties and privileged user access controls, or masking personally identi- 
fiable customer information during cloud-based testing to demonstrate 
compliance. 


Note By shifting security controls left into development and continuous delivery, it becomes 
easier to demonstrate compliance against a broad range of regulations (e.g., Federal Security 
Information Management Act—FISMA and General Data Protection Regulation—GDPR). High 
costs and delays resulting from auditors coming late into the process and finding the system isn't 
compliant may also be avoided. 


As applications become increasingly complex and threats more pervasive, 
highly skilled security specialists will become highly prized and critical 
element to the success of any DevOps initiative. Organizations shouldn't 
make the mistake of assuming that developers themselves with a smatter- 
ing of web application security experience can take on a full time security 
role (or will even want to), or that security staff (more used to maintaining 
security in legacy applications that infrequently change) can suddenly think 
like an agile developer. Over time, these skills will need to be developed 
by leveraging DevOps style collaboration. This could include agile teams 
inviting security to participate in user story development, stand up meet- 
ings, and retrospectives. For security professionals, it also means gaining 
credibility with a more detailed understanding of modern coding prac- 
tices, providing faster feedback, and becoming an active voice in all security 
related discussions. 
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Rethinking Security Practices for DevOps 


For many organizations, embedding security professionals into DevOps teams 
isn't practical. There just aren't enough of them and security operations 
may have problems scaling to handle a sudden influx of software changes. 
Addressing this requires a radical rethink on how to best apply security prac- 


tices. This can involve: 


Using security as a guiderail—Security must take a lead in 
developing solutions and policies that all development 
teams can adopt. However, if teams gain management 
approval to bypass a policy because it slows them down, 
or the business unit accepts the trade-off is well worth 
the risk, security shouldn't stand in their way. Rather, 
security should measure their security capabilities and 
continue to inform teams about the risks of their actions. 


Building closer collaboration with suppliers—With appli- 
cations moving to the cloud, organizations must work 
closely with software and cloud service providers to 
instruct what additional security controls and methods 
are needed in order to develop, test, and store informa- 
tion, without carrying additional risk. Businesses operating 
in different industry verticals will have specific compli- 
ance and data protection mandates, meaning providers 
must be willing to act in a more enterprise-friendly man- 
ner. This involves service providers including customers 
in development roadmaps and a willingness to support 
enterprise specific security requirements. 


Making security a whole-of-business issue—The role of secu- 
rity in DevOps should be to make security everybody's 
issue, not just the responsibility of the highly specialized 
security team. One effective way to do this is to develop a 
hierarchical security scoring system. If for example a defi- 
cient security practice is detected during the provisioning 
of a test environment, then the team responsible should 
be rated accordingly. That score should also bubble up to 
a group or divisional level. In this way, everyone (includ- 
ing senior management across business and IT) becomes 
more accountable. 


Proactively involving security—While it might not be prac- 
tical to embed security specialization into every team, 
the security group can establish small teams charged 
with continuously testing security across the software 
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development pipeline. At regular times this team will 
focus on particular services (even groups of people) and 
use their expertise to hunt out vulnerabilities and log 
via established ticketing mechanisms with the appropriate 
classification and priority. During these exercises, no one 
should be immune from investigation or the activity lim- 
ited to static systems. Even if teams are in the middle of 
a large important release, any severe problems detected 
must be addressed immediately. 


Essential Characteristics of Security-Minded DevOps 


As illustrated in Table 8-l, a security-minded DevOps program transcends 
beyond reacting and fixing security problems to protecting the business as 
applications are designed, developed, and tested. 


Table 8-1. DevOps and Security: Organizational Mindset 


Customers first Mindset that enables security to be continually tailored 
according to customer needs and business outcomes. 


Team alignment Flexible organizational structures that enable security 
expertise to be embedded within development, testing, and 
operational functions. 


Proactive engagement Constantly assessing the security readiness across the 
software development lifecycle by introducing unplanned 
security events and threats. 


Continuous investigation Thorough analysis of external attempts to attack a business so 
teams can remediate security issues quickly and effectively. 


DevOps and IT Service Management 


While DevOps as a movement is relatively new and many organizations are 
in the early stages of adoption, most have heavily invested in more established 
methodologies and practices—especially ITIL®'. Since its inception in 1994, 
ITIL has been positioned as the most complete approach to IT management, 
with the exception of project management and enterprise architecture. It’s 
not surprising then that upwards of two million people had some form of 
ITIL training (from foundational to expert) and that most enterprises have 
adopted many of the processes as detailed across the five ITIL volumes (ser- 
vice strategy, service design, service transition, service operations, and con- 
tinuous improvement). For some, this starts and ends with service operations 


'ITIL® is a (registered) trademark of AXELOS Limited. All rights reserved. 
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processes (especially the service desk function together with incident, prob- 
lem, and change management problems), while others have embraced a fuller 
lifecycle-based approach to adoption. 


DevOps and ITIL 


Despite common misconceptions, ITIL is not specifically opposed to agile and 
DevOps thinking. For example, the service strategy volume promotes the 
notion of continuous improvement via feedback across the service lifecycle, 
while service design mentions agile and iterative design. However, despite the 
synergies, the general philosophy behind ITIL is one of rigorous sequential 
planning and control via process; opposite of the fast iterative design approach 
of agile development. ITIL also suggests that silos will continue to exist (albeit 
aligned around 26 processes), whereas the idea of smaller cross-functional 
style product teams, fast feedback, managing work in process, and small batch 
sizes is not well supported. 


This appears to suggest that ITIL is becoming increasingly irrelevant with the 
practices under increased enterprise scrutiny. While this is perhaps true, it’s 
important to appreciate where coexistence is practical and the immense value 
established ITIL processes can still deliver a DevOps program—especially 
core ITIL processes (e.g., incident and problem management). 


Take problem management for example. This can provide considerable insight 
into the behavior and performance of a particular application, which can 
inform developers of needed non-functional improvements. This can also help 
teams avoid getting information (often conflicting) from a variety of sources. 
Additionally, a standard incident/problem ticketing method across develop- 
ment and operations may help improve teamwork and collaboration. 


Overcoming Resistance 


The biggest problem with respect to DevOps adoption is the resistance to 
change by established ITSM practitioners within the organization. Very often 
many roles are aligned around ITIL processes (change managers, problem 
managers, etc.), so it’s natural people may resist DevOps if they feel their 
careers are threatened. To this end, DevOps and ITSM leaders must actively 
work to better understand how existing roles and practices can be enabler of 
successful DevOps capabilities and where refinements are needed. This could 
include: 


Developing a better understanding of change—The adoption of ITIL has 
led many organizations to create the often-maligned Change Advisory Boards 
(CABs). Meeting infrequently, these groups have the unenvious task of approving 
production changes—essentially becoming the control point for delivery. 
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DevOps' approach to change is radically different. With DevOps, all change is 
encouraged unless it introduces greater risk and the increased probability of 
adverse customer and business impact. To this end, change isn't policed at the 
end of the cycle, but managed at the start of development. This is supported 
by many automated methods, including: 


e Automatically establishing tests early, even when require- 
ments are being established 


e Maintaining consistent environmental configurations 
across development, test, and production 


e Automated construction and invocation of testing during 
development, application builds, promotion, etc. 


e Visibility and automation of complete release workflows 


In dynamic agile environment, the rigor associated with CAB style change 
management may be too inflexible. It’s important that ITIL roles readjust pro- 
cesses to accommodate the more continuous introduction of change. This 
could involve CAB leadership determining which DevOps-related changes do 
not require the normal process rigor and may bypass the traditional controls. 


Integrating existing ITIL processes with DevOps—DevOps practitio- 
ners should collaborate with ITIL process owners to determine where inte- 
gration with existing systems can drive improvements. Even if changes have 
to go through an approval process (e.g., for systems subject to compliance 
controls), this can be streamlined by integrating release automation with the 
change request process (normally maintained in a help desk system). Rather 
than rely on manually entering a change request separately, this activity would 
be incorporated in the actual release automation workflow itself, together 
with all necessary approval requests and escalations. In another example, 
security managers could work closely with DevOps practitioners, establishing 
technologies such as privileged access management to better accommodate 
the needs of developers without compromising strict compliance controls. 


Participating in DevOps discussions—With many years of experience, ITIL 
practitioners can provide DevOps practitioners essential knowledge needed to 
drive improvements. An experienced IT operations manager could, for example, 
demonstrate how application performance management tools can be used in 
pre-production to identify any performance related problems undetected dur- 
ing earlier stages of testing. On the flip-side, ITIL process owners may ben- 
efit from newer tools introduced by agile and DevOps practitioners. Examples 
include configuration management and release automation solutions. 


Beyond discussions and as illustrated in Table 8-2, there are many other 
practices that teams can adopt to reinforce the value of more collaborative 
behaviors. 
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Table 8-2. Example Behavioral Practices for DevOps and ITIL Teams 


Practice Example 


Celebrate successes Service managers present the business outcomes from a 
successful release together with lessons learned. 


Improvement indicators Service management team attends daily stand-up meetings and 
presents how data sharing has improved the supportability of a 
new application. 


Recognition and rewards At a joint weekly meeting, DevOps and service management 
teams jointly recognize individuals who have demonstrated 
strong collaboration and a shared commitment to driving 
improvements. 


Reinforcing vision At a weekly meeting, IT operations leadership reinforces the 
vision and goals of the DevOps program. 


Satisfaction snapshots Process owners are regularly engaged to assess the current 
level of engagement and support for a DevOps program. 


DevOps and Lean Startup 


While DevOps will need to coexist with and leverage traditional IT prac- 
tices such as ITIL, there may be occasions when DevOps practices are very 
appropriate to help drive the adoption and success of newer business-driven 
methodologies. One such example is Lean Startup, a method for developing 
businesses and accelerating product development through a combination of 
hypothesis-driven experimentation, iterative product releases, and validated 
learning. 


Originally proposed in 2008 by Eric Ries?, the Lean Startup promise is to help 
businesses iteratively develop products to meet customer needs, while reduc- 
ing market risks and avoiding heavy project funding and expensive product 
launches and failures. 


DevOps practices have many synergies with Lean Startup, including: 


e Customer focus—Lean Startup places great emphasis 
on validated learning, which is basically the process for 
understanding quickly what a customer actually needs 
or wants so that useful (and profitable) products can 
be developed. Agile and DevOps can help achieve this 
through the delivery of smaller units of work or itera- 
tions, after which results are validated. 


“The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation To Create 
Radically Successful Businesses," Eric Ries, September 201 I 


DevOps for Digital Leaders 


e Fast feedback—Lean Startup success depends heavily on 
continuous customer feedback during product develop- 
ment. This is to ensure that business don't invest unneces- 
sary time and money developing features that customers 
don't want. DevOps supports this through two practices. 
First, by teams aligning activities to business outcomes and 
developing actionable metrics (discussed in Chapter 3), 
and secondly by employing continuous delivery processes 
to quickly release and test prototypes, experiments, etc. 


e Team collaboration—With Lean Startup promoting the 
continuous running of experiments and validation, it is 
critical that all cross-functional teams operate in unison. 
In such fast-paced environments, any form of waste like 
long cycle times and delays hampers validated learning 
and inhibits the flow of value. It's why DevOps prac- 
titioners supporting a Lean Startup model place great 
emphasis on reducing all elements of waste across the 
software lifecycle (see Chapter 3). 


Summary 


Being myopically focused of DevOps to such an extent that existing bodies of 
knowledge and best practices are shunned or ignored is a recipe for disaster. 
That said, organizations must also be ready to adjust existing methods, roles, 
and practices. As this chapter illustrates, this includes enterprise architecture 
and information security. 


In the next chapter, we'll describe important tangible business benefits and 
strategies needed to accelerate DevOps ROI. 


CHAPTER 
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During the 2016 Formula | season, it’s not unusual for pit-stops to hover 
somewhere around the three-second mark. In rare instances, such as for the 
Williams FI team at the 2016 European Grand Prix, the feat has amazingly 
been accomplished in less than two ticks of the clock. 


Wrap your mind around that idea for a moment. It will likely take you as long 
to digest the concept, and it certainly took longer to type out this sentence, 
than the amount of time required for a 15-20 member pit-crew to pounce on 
a stopping race car, lift it off the ground, swap four wheels and tires, and then 
drop it down to be sent on its way. Two seconds!' 


Quality Is Everyone's Responsibility 


Of course, this larger performance represents a complex chain of lesser 
events, mapped out and practiced relentlessly by highly-paid, full-time profes- 
sionals—pit-crews whose sole job is to save their teams and drivers precious 
seconds that could result in a change in position, and in some cases even a 
victory. 


"Williams Data Shows Baku Pit Stop a New Record,” Motorsport.com, June 22, 2016: 
http: //www.motorsport.com/f1/news/williams-data-shows-baku-pit-stop-a-new- 
record-790776/ 
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As one crew member pirouettes away with a used tire, another immediately 
spins into place and mounts a new wheel on the car's hub, with a pneumatic 
air gun operator barely removing the apparatus as one wheel is removed and 
another arrives in its place. This process is carried out in concert simultane- 
ously on all four wheels of the car. Suffice it to say, to pull this act off success- 
fully, execution must be flawless, every time. 


Consider that over the 70-plus years of FI history, at least ten individual races 
have been decided by less than two-tenths of a second, and many more by a 
matter of mere seconds; the impetus to strive toward such lofty speed and 
precision quickly comes into focus.’ 


Making a Case for DevOps ROI 


How do these FI crews perform such a precise ballet? By using teamwork, 
communication, and near constant review meant to spur process improve- 
ment. Countless hours are spent reviewing pit-stop video looking for even 
the smallest opportunities for potential refinement, followed by an endless 
cadence of hands-on practice wherein these movements are repeated hun- 
dreds of times. 


Meanwhile, revised wheel, hub, and pit-crew equipment designs are intro- 
duced in the name of further speeding this process. The drive for perfection 
is continuously evolved as part of a cycle that never ends. 


The return-on-investment (ROI) for teams that master this craft is evidenced 
not only in race results and unofficial pit lane bragging rights, both of which 
carry weight, but more importantly in hard dollars and cents. While races are 
often decided by seconds, the difference in prize money between the first 
place and tenth place finishers in the 2016 FI Constructors Championship is 
estimated at roughly $150 million.? 


Pit-stops represent only a small element of the overall competition, but when 
it comes down to it, every single aspect of Fl racing is aimed at shaving sec- 
onds, and even milliseconds, in the name of victory. 


Inside the modern software factory, workers are engaged in a similar high- 
stakes race aimed at addressing an elusive opportunity, which, like a race or 
season win in Fl, can disappear in the blink of an eye. This is crucible —suc- 
ceeding in the applications economy also demands constant review, improve- 
ment, and acceleration of velocity. 


?"Who Won it? 10 of FI's Closest Finishes,’ FOX Sports, September 1,2014:http: / /www. 
foxsports.com/motor/story/who-won-it-10-of-f1-s-closest-finishes-090114 
“Formula | Prize Money 2016 (Constructors Championship);" Total Sportek, March 17, 
2016: http://www. totalsportek.com/f1/formula-1-prize-money/ 
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As outlined in the proceeding chapters, it’s widely accepted that in order to 
grow and survive in the current business environment, organizations must 
embrace digital transformation, thereby advancing their ability to evolve soft- 
ware and services to meet changing customer expectations. Beyond the grow- 
ing emphasis on this theme by numerous business and technology leaders, and 
certainly speculation on the part of technology vendors, hard data has begun 
to emerge which serves to cement the requirement for DevOps adoption. 


For instance, in a 2015 study published by CA Technologies and Zogby Analytics, 
68 percent of consumers noted that they will completely abandon a particular 
brand based on mere seconds of delay in application loading times.^ Specifically, 
respondents indicated an expectation that applications load in six seconds or 
less, or else they simply drop the app, and often the related provider. Further, 
more than half these respondents indicated that they actually expect applica- 
tions loading times of less than three seconds before they run out of patience. 


Based on this reality, the need for broader adoption and advancement of 
DevOps culture, process, and tooling to speed and improve applications 
delivery is thrown into stark relief. An organization's ability to wrap its arms 
around DevOps has become a key differentiator in competing for end users' 
attention and dollars; in contrast, failing to meet changing customer expecta- 
tions related to applications quality and performance often means losing out 
to more agile competitors. 


In short, improvement of software delivery has become a prerequisite for survival. 
Based on that conclusion, beyond the vague notion of advancing general viability, 
measurement of DevOps ROI impact is clearly an increasingly critical practice. 


The Challenge of Measuring DevOps Success 


Measuring the impact of a process as far-reaching and transformational as 
DevOps within the context of ROI may seem impossible to some observers, 
but organizations of all kinds—from practitioners to industry analysts—have 
begun to attempt such calculations. 


For starters, creating the necessary framework for such metrics requires 
closer consideration of the very reasoning behind DevOps adoption. Loosely, 
this rationale is accepted as the need to increase the pace of software devel- 
opment and release, all while improving the quality of resulting applications. 
Just as importantly, DevOps also brings with it the promise of speeding the 
rate of response to emerging performance issues, all while better addressing 
changing customer expectations. 


^'Software: The New Battleground for Brand Loyalty" CA Technologies and Zogby 
Analytics, March 4, 2015: http://rewrite.ca.com/us/articles/application- 
economy/research-study-software-the-new-battleground. html 
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At first glance, this would seem a set of processes and concepts easily tied 
to straightforward notions of measurement, such as the rate of software 
deployments. However, that in itself may be too simple a viewpoint. The real- 
ity is that such measurements highlight some of the immediate benefits of 
DevOps advancement, while many other intrinsic results are certainly harder 
to quantify. 


Consider that some experts, such as DevOps industry pundit and former 
research analyst Michael Cote, have posited that attempting to gauge the ROI 


of DevOps, in this dualistic sense, is "simultaneously absurd and important"? 


The reason being, the expert maintains, that there are too many variables 
involved in DevOps transformation to make any overarching estimate of 
impact a solely believable value. 


Writing for FierceDevOps in 2015, Cote specifically noted that "DevOps is an 
immeasurable process with respect to ROI, because its value is nearly impos- 
sible to measure independently and precisely." 


That said, Cote relents that while the larger import of DevOps may be dif- 
ficult to define in terms of pure ROI, it is possible and practical to measure 
the success of specific projects—or products developed using DevOps pro- 
cesses—to establish some baseline for related assessment. This is where 
calculations such as software deployment rates and so-called mean-time-to- 
repair (MTTR)—which measures the ability of organizations to identify and 
repair applications issues that arise in production—infer some guidelines for 
assessing DevOps returns, if not larger success. 


As noted by industry watcher Christopher Null of TechBeacon in his semi- 
nal article, "How to Measure DevOps ROI,™, such measurement of impact 
is most practical within the most basic context of DevOps adoption, namely, 
saving any amount of time previously spent building, deploying, and maintaining 
applications. 


"One of the biggest benefits, if not the biggest benefit, of DevOps is the prom- 
ise of speed; DevOps enables more and faster code deployments, which means 
a decreased time to market and more opportunities to capture revenue from 
customers," Null observes. 


Using such a framework, one can also begin to tie delivery of mission-critical, 
or even revenue-impacting functionality to applications lifecycle improve- 
ments garnered via DevOps processes. For these reasons, the most practical 
forms of DevOps ROI measurement available today do in fact revolve around 
factors such as software release cycles, MTTR, and applications code change 


*'DevOps ROI,” Fierce DevOps, July 21,2015:www. fiercedevops.com/tags/devops-roi 
“DevOps ROI,” Fierce DevOps, July 21, 2015: www. fiercedevops.com/tags/devops- 
roi-how-measure-guide 
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failure rates. These are metrics that may not conclusively outline the larger 
impact of the overall movement, but that can infer some notion of ongoing 
improvement. 


While additional forms of DevOps ROI assessment are being created, includ- 
ing those focused on measurement of financial impact, enhanced produc- 
tivity, and other indicators of organizational efficiency—for now, the most 
accurate metrics are those largely focused on applications lifecycle excel- 
lence and these figures do provide a compelling case for DevOps adoption 
and advancement. 


As Null points out, Patrick Debois, himself a sysadmin, notably stated regarding 
examination of overriding DevOps value: We should be thinking about [ROI] 
in terms of ... accelerated benefits realization, and shortening that [cycle]. It's 
really not the ROI of DevOps, it's really more that the ROI of your original 
project can be realized sooner" 


A Real-World Model for DevOps ROI 


As widely accepted that applying specific ROI calculations within the domain 
of IT, in general, has always been a tricky science. This certainly applies to the 
domain of DevOps. 


This is largely related to the fact that technology has been largely perceived as 
a cost-center versus a driver of profits, making it difficult to convince manage- 
ment to increase investment with the specific goal of improving the bottom 
line. 


Even with scads of industry analyst calculations aimed at addressing this very 
issue (IT ROI), there's been little concrete evidence establishing that IT invest- 
ments can in fact provide significant returns. Most experts would more readily 
accept that IT may become a larger channel for, or catalyst to growth of the 
business, but any notions or promise of massive ROI is often overshadowed 
by requirements for sizeable upfront and ongoing investment. 


Further, technology vendors notoriously muddy the waters by proffering 
self-serving calculations that often inflate the proposed impact of their 
products in the name of boosting sales and competitive prospects. That 
said, it's also recognized that, especially in recent years, technology para- 
digms such as mobility, the cloud, and virtualization have delivered econo- 
mies of scale and new business opportunities so massive as to make them 
difficult to fully quantify. 


So, how does one go about making a believable ROI case around DevOps, a 
process that, like many other movements, promises to completely redefine 
the manner in which organizations create and monetize their value? 
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The 2016 State of DevOps Report, the industry's leading research proj- 
ect—conducted and prepared by the DevOps Research and Assessment 
(DORA) initiative—offers an avalanche of data providing concrete testa- 
ment to the proposed, and oft-considered fuzzy, benefits of the overall 
movement." 


For starters, the report posits that so-called "high-performing organizations," 
those already deeply engaged in the use and maturation of DevOps prac- 
tices, are "decisively outperforming their lower-performing peers in terms 
of throughput.” This finding not only builds on similar conclusions in previ- 
ous iterations of the same report, but also maintains that the espoused gap 
between DevOps adopters and laggards continues to widen. 


Specifically, DORA contends that such high performers deploy applications 
code 200 times more frequently than organizations that have not yet closely 
embraced DevOps. So, practically speaking, such organizations have seemingly 
become far more capable of engaging the benefits of agile computing; namely, 
responding adeptly to changing customer demands. 


In addition, the State of DevOps Report also finds that high-performing orga- 
nizations are currently achieving 2,555 times faster lead times, serving as even 
greater proof of their abilities to remove obstacles to software innovation and 
improvement. 


Perhaps even more importantly, in particular as it relates to the matter 
of agility and the ability to cater to evolving performance conditions, the 
report estimates that leading DevOps practitioners lay claim to 24 times 
faster recovery times and three times lower change failure rates. These 
figures formulate a highly compelling narrative when it's recognized that 
reduction of MTTR represents one of the most critical differentiators in 
today's applications economy. 


The State of DevOps Survey, completed by roughly 4,600 technology practi- 
tioners, also found that high-performing teams spend far less time on so-called 
“unplanned work and rework"—most often required to respond to emerg- 
ing issues related to existing applications code. This allowed them to spend 
nearly 50 percent more of their time on new work. The latter brand of effort 
is typically aimed at adding or improving feature functionality in the name of 
increasing business opportunities. 


Lastly, in terms of pure applications delivery ROI metrics, the State of DevOps 
Report suggests that leading practitioners are suffering far fewer applications 
and services outages—those timeframes when systems are offline and truly 
prevent the flow of end user traffic and business. 


™2016 State of DevOps Report," DORA and sponsors, June 22, 2016: https: //puppet. 
com/resources/white-paper/2016-state-of-devops-report 
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Based on a framework established by research analysts at IDC, which concludes 
that hourly application downtime costs can range from $1.25 to $2.5 billion for 
a Fortune 1000 firm, and that the average cost of a critical application failure is 
$500,000 to $1 million per hour,’ the State of DevOps Report allows for the con- 
clusion that high-performing organizations, compared even to those with some 
level of DevOps adoption, save an average of over $91 million per year. 


Much of this advantage is found, the researchers note, in the form of lower 
applications change failure rates, or those updates to software code that 
result in subsequent performance issues or outages. While high-performing 
DevOps practitioners experience an average change failure rate of 7.5 per- 
cent, medium performance organizations see that figure rise dramatically, to a 
whopping 38 percent. 


If the undeniable adage that "time is money" applies in today's applications 
economy as much, if not more than ever, then these ROI calculations carry 
substantial meaning. 


Beyond such metrics aimed at measurement of improvement appreciated 
across the applications lifecycle, the 2016 State of DevOps Report also sheds 
light on another area of the movement's impact that would appear to tran- 
scend immediate calculation, but which can certainly be quantified in general 
terms. Organizations deeply engaged in DevOps transformation tend to have 
happier, more dedicated employees. 


Given the wide recognition that it is far more expensive to hire and train 
new employees than retain existing staff, and that highly skilled IT workers in 
particular are difficult to find and hold on to, this less hands-on ROI benefit of 
DevOps may prove as even more critical support for adoption than some of 
the proceeding metrics. 


Specifically, DORA's researchers found that employees of high-performing, 
DevOps-centric organizations are 2.2 times more likely to recommend their 
organization as a great place to work, and 1.8 times more likely to recom- 
mend their team as a great working environment. The figures were calculated 
using the employee Net Promoter Score (eNPS), which is designed to mea- 
sure overall employee loyalty. 


As a follow on, it's worth noting that other research, notably a report pub- 
lished by experts at Bain & Co., has found that companies with highly engaged 
workers typically produce revenues 2.5 times greater than those with low 
engagement levels. In addition, Bain submits that publicly traded stocks of 
companies with a high-trust work environment outperformed market indexes 
by a factor of three.’ 


“DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified,” 
IDC, December 1,2014: http://www. idc.com/getdoc. jsp?containerId=253155 
*http://www.bain.com/publications/articles/the-chemistry-of-enthusiasm. 
aspx#2 
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Simply stated, these conclusions, both process-specific and more general in 
nature, highlight the specific effect that DevOps is having in terms of creating 
the most critical form of ROI measurement—the ability to innovate faster, 
build organizational momentum, reduce costs, and attempt to differentiate 
offerings within the competitive environment. 


It only follows naturally that organizations spending more of their time creat- 
ing new value by translating ideas into product offerings are likely outperform- 
ing those struggling to play catch-up by addressing existing shortcomings. 


Measuring the Impact of DevOps Automation 


As previously noted, success in every element of the Fl racing arena is mea- 
sured in the smallest of increments, in this era most often in the form of 
tenths, or more typically hundreds of seconds. For FI pit-crews, refinement 
of each tiny physical movement or adaptation of time-saving technology can 
result in a measurable benefit. 


The same can be said of DevOps adoption, which can be extrapolated to span 
nearly every aspect of software design, delivery, and support, along with a 
multitude of tools aimed at automating some involved process. 


To isolate any one or two specific processes or tools only provides a small 
window into the broader impact of DevOps automation as a whole; however, 
leveraging these as a starting point can provide perspective which can then be 
expanded across numerous domains. 


One such toolset that has been so examined is the utilization of CA Agile 
Requirements Designer solution, which seeks to automate test case design 
and execution based on changing requirements—to automate one of the 
most time-consuming and resource-intensive problems in applications deliv- 
ery—namely, delays created by manual testing. 


By allowing engineering teams to automate the process of creating and apply- 
ing software testing requirements, the solution promises to greatly reduce 
related cost and complexity, specifically by directly cutting down the number 
of required test cycles. 


According to a calculation designed to forecast the potential ROI of CA Agile 
Requirements Designer related to cost reduction and avoidance, along with 
related revenue enhancement, immediate benefits of employing this process— 
representing only one minor aspect of DevOps adoption—immediately come 
into focus. 


Based on a real-world implementation of the solution, in addition to gains 
related only to software testing, one organization found that it was able to 
drive far closer collaboration between business and IT teams, and better 
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estimate the time and cost of development projects. In addition to the fol- 
lowing ROI calculations, these gains map closely to core benefits of broader 
DevOps adoption.'? 


In terms of specific ROI, the involved organization found that its worst case 
appreciation of benefits over a three-year period would result in a 168 per- 
cent return on its existing investment, or a period of || months before 100 
percent realization of payback on initial costs; the study projected a best case 
scenario of 319 percent ROI, or 8 months for complete payoff. 


Among the underlying benefits driving this return were the organization's 
ability to reduce software defect remediation (a similar domain to reduction 
of MTTR), lowering of test creation and maintenance costs, and, most impor- 
tantly within the overall scope of strategic investment, improved time to value 
for application releases. 


At the production end of the software lifecycle, another important DevOps 
tooling aspect that warrants ROI analysis is Application Performance 
Management (APM). APM seeks to simplify the triage of performance prob- 
lems so teams can address issues fast, before there's an impact on end users 
and customers. 


By quickly analyzing performance across a complex array of microservices, 
APIs and containers across mobile, cloud, and on-premise applications, these 
monitoring tools provide IT operations an essential mechanism for main- 
taining expected levels of service together with a high-quality application 
experience. 


In a DevOps context and as described in Chapter 7, these tools are especially 
valuable because they help establish feedback loops between operations and 
development teams. By gaining insights into application performance before 
committing software updates to production, developers can quickly pinpoint 
and remediate any issues related to their code. In this way, APM solutions not 
only help ensure operational stability, but also assist teams in building quality 
into the applications. 


According to a 2015 Total Economic Impact study conducted by Forrester 
Research, the potential ROI of CA APM is related to significantly reducing 
downtime costs and improving productivity.'' 


“CA Test Case Optimizer ROI Business Case Detailed Report,’ CA Technologies, July 
1, 2015: https: //km.ca.com/sales/presales/roi/ROIZ20Portal/Dashboard/CAX20 
Test%20Case%200pt imizer“20Reference%20RO1I%20Bus iness%20Casex20V1FY16REF. 
pdt 

The Total Economic Impact of Application Performance Management: http://www. 
ca.com/content/dam/ca/us/files/industry-analyst-report/the-total- 
economic-impact-of-apm.pdf 
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Interviews with four customers (across financial services, banking, and health 
insurance sectors) and subsequent financial analysis revealed that a composite 
organization based on the interviewed organizations experienced benefits of 
more than $7.2 million versus implementation costs of just over $1.7 million, 
adding up to a net present value (NPV) of $5.4 million and an ROI of 306 percent. 


As would be expected, all the interviewed organizations experienced a posi- 
tive financial impact by improving the availability and performance of customer- 
facing applications. For one company, this extended to reducing service-level 
penalty fees because it was able to prove the cause of problems to a key com- 
mercial customer, reducing the flow of more than $800,000 that it was paying 
yearly to a large, global vendor. 


Beyond availability gains, and importantly from a DevOps perspective, there 
were other benefits. The study found that by applying CA APM within the 
development process, developer productivity increased. When the develop- 
ment team used CA APM to test new code, the organization was able to 
accelerate development cycles by 15 percent in the first year. As a representa- 
tive of one company said,"In essence, we're issuing less code or less fixes into 
production, which allows our developmental organization to focus more on 
maintaining a competitive advantage in the marketplace of what we deliver in 
our online solutions." 


Leveraging CA APM provided insight into performance from mainframes 
through mobile devices. Using CA APM allowed the organization to elimi- 
nate uncertainty about the cause of problems in complex environments. 
CA APM also reduced the average time to resolution and it eliminated the 
finger-pointing. 


Avoiding blame games and a hero culture is of course the essence of DevOps, 
but application complexity and needing to support multiple stakeholders 
means tools must be capable of delivering uninterrupted visibility. The study 
illustrated that by providing performance insight from mobile to mainframe, 
CA APM allowed the organization to eliminate the uncertainty about the 
absolute cause of problems in complex environments. As one performance 
analyst from a regional consumer bank stated, “APM gives us micro-visibility 
into the customer experience.” 


The Whole Is Greater than the Sum of its Parts 


Given that these aspects (agile requirements design and APM) of applications 
delivery represent only parts of the overall lifecycle and areas for potential 
DevOps impact, one can easily justify adoption based on the ROI they singu- 
larly deliver. However, to be truly reflective of DevOps impact, tools should 
be examined in unison. This way, organizations can begin to extrapolate how 
combinations across the automated software factory can create the massive 
gains highlighted in the State of DevOps Report. 
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Figure 9-| and the notes that follow illustrate a simple but powerful example 
of the cumulative benefit of combining automated tools across the software 
lifecycle. 


995» Ri ) B O 


REQUIREMENTS DEVELOP / TEST PRODUCTION APPLICATION 
TEST CASE & SUPPORT EXPERIENCE 
DESIGN ANALYTICS 


Figure 9-1. Cumulative benefits of combining multiple tools example: application experience 
analytics and agile requirements design 


Step |: Application Experience Analytics 


In this step, tools feed information back to agile teams, which is used as input 
for requirements changes and iterative development. 


Expected benefits include constant cycling of intelligence gained through ana- 
lytics (e.g., app and functional usage and live performance) that helps develop- 
ers, operations, and the business improve applications with every release. 


Step 2: Requirements and Test Case Design 


As requirements change, agile requirements design tools automate test case 
design and their execution. 


Expected benefits include maximum functional coverage with the small- 
est number of tests, as well as reduction in test case cycles by automatically 
removing any outdated, redundant, or duplicate test cases. 


When other tools are integrated, the benefits will continue to accumulate. For 
example, by automatically providing synthetic data from a test data warehouse 
and matching this directly to the test cases created in Step 2, an organization 
can: 


e Improve quality—Providing the right data for testing can 
reduce defect creation 


e Improve testing efficiency—Eliminating data constraints can 
reduce time and resources needed to provision data 
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e Reduce costs—Creating accurate subsets of data can 
reduce infrastructure cost 


Summary 


As you've seen, current measurement of DevOps ROI remains somewhat 
nascent, and difficult. However, real-world use cases and industry research 
clearly highlight the massive impact that the adoption of DevOps practices is 
having among today's early adopters. The future looks bright for those who 
get on board, while those left behind may struggle to compete and survive. 


In the next and final chapter, we'll look at some additional concepts, techniques, 
and emerging trends that have begun to populate the DevOps landscape and 
examine how these may be used to enhance a DevOps program. 


CHAPTER 


10 


DevOps 
Finetuning 


Additional Considerations, Concepts, and Practices 


Congratulations, you've read this far and are ready to floor the DevOps accel- 
erator. Like a Formula | driver in pole-position at the start of a race, you're 
eager to get the green light and hit the gas. Hopefully, you're part of a team 
with a winning culture, managing to business outcomes, and building the strat- 
egies needed for continuous improvement. 


But this is no time to be complacent. Like Formula circuits and race day 
conditions, business changes constantly. Just as the Monaco circuit demands 
a different race strategy than Monza, DevOps programs must adjust to 
different circumstances. For a business focused on operational efficiency 
(doing more of the same, only faster and cheaper), or government agencies 
with shrinking budgets, Lean aspects of DevOps might take precedence. 
But if a business is adjusting its operating model (doing the same things, 
but in radically different ways), agile methods and continuous delivery will 
come more to the fore. Whatever the case, be prepared for change and 
never underestimate the importance of strong culture and building and 
cultivating great teams. 


© CA 2016 
A. Ravichandran et al., DevOps for Digital Leaders, DOI 10.1007/978-1-4842-1842-6 10 
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Cultural Recalibration 


Formula driver Daniel Ricciardo drove brilliantly at Monaco in 2016, but 
when his pit-crew wasn't ready for a tire change, he lost precious seconds 
and the race. And while ten seconds or so doesn't seem like much, it made 
the difference between Ricciardo spraying champagne and finishing a mis- 
erable second. 


Incidents like these reaffirm that people make mistakes all the time—it's what 
makes us human. Even a finely-tuned DevOps program won't be immune to 
people-related issues and why culture can't be overstated. 


Unlike process, culture can't be automated (not yet anyway), which is why 
organizations must constantly work on fine-tuning.Yes, this involves building a 
strong collaborative workforce, but also understanding what conditions exist 
that cause people to persist or revert back to behaviors counter to DevOps 
thinking-also known as "falling off the DevOps wagon". 


The Normalization of Deviance 


How many times have you witnessed a sub-optimal IT practice that every- 
one else thinks is okay? Then over time have you accepted the behavior and 
started practicing it too? We all have—it's quite normal. 


Regardless of whether you lead a startup or work in an established business, 
we all have a tendency to accept suspect behaviors. Even if outsiders see them 
as wrong, our [T teams are so accustomed to using them (without any adverse 
consequences) that they're quickly established as "normal" and accepted. 


Studies into what's commonly referred to as the "normalization of deviance" 
have been conducted in areas from healthcare to aerospace, with evidence 
showing that many serious errors and disasters occur because established 
standards have been bypassed and bad practices "normalized". 


While examining this phenomena is critical in the context of safety, it's 
equally applicable in how we develop, secure, and operate software applica- 
tions. With the boundaries blurred between the digital and physical world, 
any adverse behavior leading to security and reliability issues could have 
dire consequences. And when software becomes infused into long-lasting 
products (from light bulbs to limousines), it's not so easy to discretely exit 
markets. 


As businesses increasingly rely on software innovation for market expansion, 
faster time-to-market and a high-quality customer experience become essen- 
tial differentiators. Unfortunately, both can be compromised if rigid change 
controls or "speed at all cost" mandates lead to a bad behaviors. 
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Interestingly, guidance from other fields can help IT identify and eliminate 
poor practices. In the healthcare field, for example, studies have identified 
seven factors that lead to a normalization of deviance.' 


All of these are extremely relatable to IT. 


The rules are stupid, dumb and inefficient—In healthcare, accidents can 
occur when medical practitioners disable equipment warning systems because 
alarms are seen as distracting. This happens in all the time in IT with dire con- 
sequence. Like when IT operations staff miss major problems because they 
filter out noise and alerts on monitoring consoles they regard as irrelevant. 
Or when testing is deliberately skipped because of lengthy manual processing 
and provisioning delays. 


Knowledge is imperfect and uneven—Employees might not know that a 
rule exists, or they might be taught a practice not realizing it's sub-optimal. In 
IT this persists because many new employees feel uncomfortable asking for 
help, or the application of new technologies distorts logical thinking. 


The work itself, along with new technology, disrupts work behav- 
iors—To support goals of more continuous software delivery, organizations 
are introducing many new technologies and methods, such as microservices 
and containers. Along with the new tech, new work practices and learning 
demands may lead staff to poorly implement the technology or use it to per- 
form a function it was never designed for. For example, containerizing a legacy 
system just because it's "technically feasible". 


We're breaking rules for the good of the business—Staff may bypass 
rules and good practice when they're incentivized on faster delivery times 
or delivering new functional software enhancements. For example, procuring 
additional (but unnecessary) hardware capacity to rush through an update, 
rather than addressing the root cause of persistent performance problems. 


The rules don't apply to us...just trust us—Empowered agile teams are 
fantastic, but with great power comes great responsibility. Teams might slam 
through more code, but poorly governed open source access or bypassing 
compliance policies when generating test data can derail a program or lead to 
massive security breaches. Unfortunately in today's fast-paced digital business, 
talented professionals often feel completely justified in playing the "trust us, 
we know what we're doing" card. 


Employees are afraid to speak up—Violations become normal when 
employees stay silent for fear of admonishment. How many times have bad 
software code practices been tolerated because junior staff is afraid to speak 
up? Even in IT organizations with a strong blameless culture, people can remain 
silent for fear of appearing "mean" to their colleagues. 


‘http: //www.ncbi.nlm.nih. gov/pmc/articles/PMC2821100/ 
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Leaders withhold or dilute findings on application problems— 
Whether you work in healthcare or IT, no one wants to look bad to managers. 
Rather than present ugly and unpleasant realities, many will distort the truth; 
presenting diluted or misleading information up the command chain. In IT, this 
behavior is easily normalized, especially if teams get away with reporting tech- 
nical vanity metrics over business outcome-based performance indicators. 


Tip Staff will never admit to anything they don't see as wrong. Get out of the office and 
“gemba,” or walk your own software production line. Look for all the warning indicators that might 
be causing people to do the wrong things. 


Combatting Sub-Optimal Behaviors 


No sudden cultural reawakening across the IT organization will eliminate 
ingrained bad practices, but DevOps and Lean thinking can help identify the 
warning signals and can become the catalyst to drive behavioral improvement. 
This can involve: 


e Fostering open discussion—lt's easy to stand back and 
do nothing if a sub-optimal practice has yet to cause 
problems. But over time, poor behaviors become more 
difficult to remove, so catching early is critical. Ongoing 
education and open discussion can help with both 
development and operations participating. Some of this 
might be uncomfortable and peer-level critiques are 
never easy, but it's better than conducting blame games, 
post outage or major incident. 


e Managing to outcomes—People are reluctant to change 
behaviors if they perceive the probability of a major inci- 
dent or outage to be low. Part of this is normal human 
behavior, but can be exacerbated when the staff down- 
grades the likelihood of problems eventuating because 
diagnostics point only to minor technology issues. 
Therefore, it's important for staff to clearly understand the 
business impact (worst case scenarios) of poor practices. 


e Finding visual clues—This includes visualizing the flow of 
value delivered by software applications and pinpointing 
all the bottlenecks and constraints. Analogous to unsafe 
stepping stones across a stream, these are the value inter- 
rupts. When examined, they reveal all the process and 


DevOps for Digital Leaders 


technology issues causing staff to do the wrong things. 
Immediate candidates are software release and testing 
processes, but analysis shouldn't be restricted to devel- 
opment. Everything from enterprise architecture, stake- 
holder engagement, and information security, to vendor 
management and customer support can be a choke-point. 


DevOps and Talent Management 


While DevOps promotes the collaboration of software development and 
other IT groups, there is nothing stopping other departments across the orga- 
nization from becoming involved in the program. Even the HR team can be 
included, especially with regards to talent acquisition and development. 


In the application economy, demand for technology professionals in areas 
such as IT architecture, big data/analytics, and cloud computing is growing fast. 
Having DevOps skills is also becoming financially attractive. As was indicated 
in a 2015 survey from the cybersecurity firm Incapsula, which revealed that 
high school graduates in entry-level jobs requiring DevOps skills are seeing 
a median starting salary of $106,000? But in advanced economies, demand is 
outstripping supply. In the UK, for example, the Tech Nation report, from Tech 
City and innovation charity Nesta, indicated that 40 percent of digital entre- 
preneurs say they face challenges finding skilled digital workers.? 


The source of skills is also changing. The Tech Nation report also revealed 
informal or in-house training as the most common source of skills, with tradi- 
tional universities ranking relatively low in IT activities.This is in no small part 
due to easier access and cheaper technology. For example, and as of July 2016, 
GitHub (the open source development platform) has 15 million people col- 
laborating across 38 million repositories, while the credit card-sized raspberry 
PI processor has reached 5 million sales. 


So You Think You Can DevOps? 


With major supply challenges, HR can work collaboratively with IT to enrich 
the organization with much-needed DevOps skills. This will involve sourcing 
by traditional mechanisms (e.g., online job boards), but since DevOps requires 
no formal qualifications or certification, careful attention during candidate 
screening and selection is needed. 


?DevOps Salary Survey, 2015: http://lp.incapsula.com/rs/incapsulainc/images/ 
Report%20-%20Dev0ps%20Salary%20Survey%202015. pdf 

?Tech Nation, 201 6: http: //www.nesta.org.uk/sites/default/files/tech nation 2016 - 
report.pdf 
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Ideally, DevOps practitioner should exhibit the following skills: 


e  People—DevOps practitioners need to be good commu- 
nicators, which is a skill that's not normally associated 
with introverted technologists. Look for people who 
can exhibit evidence of written and verbal communica- 
tion—within, across, and external to organizations. For 
example, has the candidate presented at conferences or 
tech meet-ups? 


Note HR can help IT develop good communication skills through the introduction of formal 
courseware and “lunch and learn” sessions. 


Careful consideration should be given to candidates who 
have experience working with business stakeholders 
and can clearly articulate strategy, priorities, and goals. 
DevOps practitioners should be able to describe situa- 
tions where they collaborated and shared information to 
achieve better business outcomes. 


e Process—While it's possible for DevOps to work with 
Waterfall development, success is more likely in environ- 
ments where agile methods have been applied. Strong 
candidates will demonstrate understanding and expe- 
rience in one or more agile approaches (e.g., Kanban, 
Scrum, or SAFe). 


Value should also be placed on candidates who can dem- 
onstrate knowledge in Lean manufacturing concepts such 
as Just-In-Time and Theory of Constraints. Extra marks 
if they can illustrate examples of where and how they 
applied these principles in an IT context. 


Note Working with IT managers, HR could introduce an online library of DevOps, agile, and 
Lean related books. Some great examples include The Phoenix Project and The Goal and Lean 
Enterprise.’ Don't limit access to IT, make the resources available across the organization. 


‘The Phoenix Project, Gene Kim, Kevin Behr, George Spafford, ISBN 978-098826259 | 
The Goal: A Process of Ongoing Improvement, Eliyahu M. Goldratt, ISBN 978-0-88427-178-9 
Lean Enterprise, Jez Humble, Joanne Molesky, Barry O'Reilly, ISBN 9781449368425 
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e  Technology—DevOps practitioners will have acquired a 
range of skills across the software development lifecycle, 
including, but not limited to, agile project management, 
build management, and release automation. Strongly con- 
sider candidates who are experienced integrating these 
to build a fully automated DevOps toolchain. 


Note Look for individuals who have worked in cross-functional teams and have broadened 
their skills by learning new technologies. Individuals who not only use technology but have 
contributed to its development are also good candidates. 


Rather than present resumes, strong technologists will emphasize their 
achievements and expertise via actual contributions within the agile and 
DevOps community.This can include technology blogs, and open source con- 
tributions (code, scripts, and commentary). Ideally, sysadmin candidates will 
have coding skills across a variety of programming languages, while developers 
will be familiar with scripting methods. 


Various Talent Management Hacks 


Age shouldn't be a factor when selecting DevOps talent. Many of the best 
practitioners have years of experience adopting new technologies and prac- 
tices. In the enterprise, this is especially valuable since many legacy systems 
will need to be integrated with cloud applications, which requires a compre- 
hensive understanding of the inner workings of many technologies and all 
their limitations. That said, however, it'll be necessary to replenish DevOps 
programs with fresh talent. And since supply is problematic, HR can assist 
technology teams with innovative acquisition and development strategies. This 
could include the following. 


IT Apprenticeship and Graduate Intake Programs 


Many organizations have graduate intake programs, some of which don't 
require applicants to have formal IT qualifications. They encourage anyone to 
apply who has a passion for technology. One example is the ICT Apprenticeship 
program run by the Department of Finance in Australia.’ After a series of 
interviews, successful applications are employed by a sponsoring government 
agency. Upon assignment, they combine four days of work with one day of 
formal IT study, leading to certification. 


http://www. australia. gov.au/information-and-services/jobs-and-workplace/ 
australian-government-jobs/ict-apprenticeship-programme 
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Such programs are a great way to infuse talent within an organization, but 
there are challenges that HR teams must address. Often in these types of ini- 
tiatives, apprentices are rotated through different areas of IT—the aim being 
to familiarize new staff with all aspects of the IT business.This is fine in theory, 
but in a DevOps context (where cultural improvement is so important), it can 
actually be counterproductive. If, for example, the organization is functionally 
siloed, divisive, and averse to change, apprentices could become isolated, and 
rotating passionate and eager technologists through highly process-centric 
areas can be the fastest way to dampen their spirit and enthusiasm. 


Tip Consider organizing graduates or IT apprentices in self-managing teams or “pods” with 
full program oversight by experienced leaders and mentors. Always assign work that is meaningful, 
valuable, and measurable. Stagger the arrival of new intakes and place them in the same workplace 
as existing apprentices. 


DevOps Hackathons and Coding Days 


Hackathons are short events where all kinds of specialists come together to 
work intensively on small technology projects. Hackathons can be run as a 
contest (e.g., prize for best mobile app innovation), hosted internally or exter- 
nally, or purely social. In some cases, they even have the lofty goal of creating 
commercially viable software solutions. 


With DevOps, hackathons can be valuable for instilling the value of collabora- 
tion and teamwork within the IT organization. HR can take the lead in helping 
to organize and arrange these events, but should work with IT leadership to 
ensure maximum DevOps benefit. For example, conducting the hackathon 
away from the workplace where participants could be distracted. Other fac- 
tors to consider include: 


e Balancing teams—lIncluding developers, sysadmins and 
project managers, enterprise architects, and security 
specialists. 


e Providing broader deliverables—Don't restrict hackathons 
to new apps. Look for other opportunities to improve 
quality. For example, give prizes for the fastest way to 
find the root cause of a complex application performance 
problem; the most innovative application of security to 
increase resilience; and the best API to securely expose 
enterprise data. 
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e Managing expectations—Don't expect clean and robust 
code from a short hackathon; you'll get alpha at best. 
Consider longer online developer challenges if the goal is 
to fuel business with viable business prototypes (e.g., inte- 
grating valuable enterprise data with third-party systems). 


Involving partner developers in a hackathon is a great opportunity to nur- 
ture innovation but requires additional oversight and support. Providing badly 
formatted open data without metadata, or poorly documented and unman- 
aged APIs, should be avoided.When involving third-party developers, consider 
methods to simplify API discovery, registration, support, and engagement, 
including providing an API catalog, sample code, and mobile device SDKs. 


DevOps and the Internet of Things 


At the 1990 Interop computer conference, two guys connected a kitchen 
toaster to the Internet. This early attempt at appliance networking paved the 
way for what's now called the Internet of Things (loT).° 


As of 2016, it's estimated there could be more than 10 billion Internet- 
connected elements, ranging from watches and cars to just about anything in 
the "thing" category, including light bulbs, fitness wearables, fridges, and, yes, 
even pets and diapers. 


But beyond the big numbers, marketing spin, and hype, the business potential 
of loT is being increasingly exploited by early movers. Perhaps not surprisingly, 
this includes some major businesses and brands. 


Bosch, for example, the German multinational engineering and electronics 
company, announced an loT cloud.This is the latest initiative as the company 
transitions from traditional industrial company to a blended manufacturer and 
technology provider. 


While connected home and car use cases grab our attention, many industries 
are exploiting standards, unified architecture (IT and industrial), and innovative 
software solutions to connect device and sensors to IT systems. 


Take for example manufacturing. When plant production systems can be inte- 
grated with customer demand analytics, maintenance windows can be opti- 
mized. Similarly in energy and utilities, integrating smart meters with back-end 
applications and analytics could become a system to predict energy demand 
across the grid. 


*http://www.livinginternet.com/i/ia myths toast.htm 
"http://www.wsj.com/articles/robert-bosch-launches-own-cloud-for- 
internet-of-things-1457528014 
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This convergence will require new approaches to developing and operating 
applications, and DevOps with its focus on shortening delivery cycles while 
improving software quality is absolutely essential. However, loT project suc- 
cess depends on DevOps practitioners addressing some new challenges: 


e Bridging another cultural divide— The development and oper- 
ations divide is nothing compared to the cultural chasm 
spanning traditional [T and industrial operations. While IT 
speaks of Java, C++, NoSQL, and the IP stack, plant and 
industrial engineers talk of programmable logic controllers 
and supervisory control and data acquisition (SCADA). 


Note Putting all IT “ego” aside, DevOps teams need to build a good understanding of loT 
related terms, technologies, and limitations. To assist, try to involve product specialists (e.g., 
industrial engineers, product managers, and designers) in DevOps/loT initiatives. 


e Solving complex security problems—Connecting previously 
isolated physical devices to networks creates additional 
threat surfaces. At best this could mean service interrup- 
tion, at worst physical injury or loss of life. This conse- 
quence was disturbingly illustrated in the Jeep security 
exploit, where hackers demonstrated wireless access to 
vehicle infotainment and controls.? These could be expen- 
sive loT lessons in the automotive industry where fleets 
of new cars can leave a dealership containing up to 70 spe- 
cialized computers and more than 20 million lines of code. 


Note Considerable investment will be needed to address loT security issues. DevOps and 
security teams must review entire loT processes (device to cloud or back-end application), thus 
ensuring that embedded software applications and data can be verified and encrypted. 


e Managing privacy—Internet privacy issues could be noth- 
ing compared to the challenges introduced by loT. As 
the physical and digital world becomes blended, smart 
devices have the potential to track behavioral patterns 
everywhere—as we sleep, during exercise, when we 
drive, consume energy—you name it! 


https: //www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/ 


DevOps for Digital Leaders 


Tapping into this information means DevOps practitioners must pay care- 
ful attention to issues around data ownership, compliance, data access, and 
sharing arrangements. 


Note Asmore customer data is obtained from loT devices and consumed by complex analytical, 
and big-data systems, careful consideration to accurate and compliant data management (e.g., 
during testing) becomes more critical. 


e Guidance on loT interoperability—Many industrial opera- 
tions and loT systems work in isolation and use proprie- 
tary networks (e.g., Modbus and BACnet) and protocols. 
In many cases, what works in one industry vertical won't 
work in another. While this problem can't easily be 
addressed, DevOps teams involved in loT projects must 
become knowledgeable on emerging loT standards and 
interoperability initiatives. Examples include The Open 
Connectivity Foundation and the Allseen Alliance.’ 


e Understanding physical device limitations—Depending on 
business context, loT devices characteristics and usage will 
impose many limitations on software development, testing, 
and operations. Not the least, constraints on what pro- 
gramming languages can be used for loT development and 
strict limitations on runtime size. Since software won't be 
operating in the confines of a 24/7 data center, there'll also 
be design issues to factor in (e.g., device battery life, envi- 
ronmental/weather conditions). Consider too, challenges 
making software updates across sub-optimal networks, the 
mass migration of sensor data to the cloud for analysis, and 
determining embedded software security risks (especially 
in OEM devices and third-party components). 


DevOps for the Public Sector 


You can be forgiven for thinking that DevOps only works in the private sector. 
It's simply not true. 


Governments across the world continue to search for better ways to deliver 
citizen services through new technologies and practices. Agile and DevOps 
are gaining traction, with federal U.S. departments, such as U.S. Citizenship 
and Immigration Services (USCIS) and the Environmental Protection Agency, 
becoming adopters. 


?https://allseenalliance.org/https://openconnectivity.org/ 
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But adoption isn't limited to the United States. In Australia, the recently 
formed Digital Transformation Office (DTO) leverages agile methods and 
DevOps principles to support its mission of leading the digital transformation 
of government services. '? 


But as of 2016, the public sector hasn't fully embraced DevOps thinking. This 
was well illustrated in a Vanson Bourne study, which indicated that public sec- 
tor organizations are a third less likely to adopt DevOps compared to their 
private sector counterparts.'' 


Yet those who are starting a DevOps journey are seeing benefits. As indi- 
cated in the study, more than one-third of public sector DevOps adopters 
have experienced application quality improvements, while almost half have 
seen reductions in IT spend.This last point is especially pertinent for budget- 
constrained government agencies. 


DevOps and Lean to Improve Government IT 
Outcomes 


Government agencies are under increased scrutiny to ensure money is spent 
wisely. Unlike their private sector counterparts, who can justify mega IT 
spending as the means to increase revenue and market share, government 
agencies can only think in terms citizen-centric service improvement, which 
given the overarching political context and climate, might not be a top priority. 


Even when budget is available, the track record in government IT delivery 
hasn't been great. According to Standish Group, just over 94 percent of all 
large government IT projects (2003-2012) were over budget, behind schedule, 
fell short of user expectations, or had to be abandoned completely. "? 


With a 6 percent success record, it's perhaps not surprising that the DevOps 
mantra of "failing fast" doesn't fully resonate in government departments. 
Apart from the cost factor, no IT manager would want to front a congressio- 
nal committee when critical government services have been disrupted. 


With increasing budget constraints, probably the best place to start a DevOps 
program in a government agency is to focus on the Lean principles that under- 
pin much of the thinking. By identifying all elements of “waste” that incur greater 
cost and negatively impact delivery, agencies can improve cost structure while 
speeding delivery. To this end, and before any investment is made in tools, 


https: //www.dto.gov.au/ 

"https: //www.ca.com/us/rewrite/articles/devops/research-report--devops- 
the-worst-kept-secret-to-winning-in-the-application-economy.register.html 
"http://www. brookings.edu/blogs/techtank/posts/2015/08/25-yaraghi- 
government-it-projects 
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mature government agencies will carefully examine the entire end-to-end soft- 
ware delivery cycle and remove any cultural, process, or technology related 
inhibitors that add no value. The intention of course is to increase speed and 
quality, but a very important “by-product” for the budget constrained agency is 
cost reduction. 


Many wasteful practices traditionally associated with manufacturing are appli- 
cable in a government software development context. 


e Long waitt—Some government agencies remain fixated on 
performing lengthy change review processes. But with new 
technologies now making it possible to deploy fully cer- 
tified stacks to the cloud, only application changes need 
to be checked. This significantly reduces release times 
and administrative costs. Of course, there will be occa- 
sions when teams require access to production systems 
and data for testing, but cycle time can be shortened with 
technologies that virtualize or emulate dependent systems. 


e Excess inventory—Over years, agencies have acquired a 
complex array of technology. These include mainframes, 
applications, and databases. Each servicing agency require- 
ments in everything from massive transactional processing 
to data matching and analytics. In order to manage this 
complexity, agencies have become reliant on contractors 
who generally shun agile and DevOps concepts, preferring 
instead contracts with fully defined requirements and lon- 
ger delivery cycles.To address this, agencies should imme- 
diately review the IT culture and skills gaps issues this 
causes, plus consider engaging nimbler agile and DevOps 
minded contractors via simpler procurement processes. 


e Slow motion—As with large commercial enterprises, gov- 
ernment agencies can be inhibited by bureaucracy and 
controls. As governments look to adopt DevOps, it’s 
important to analyze every unproductive process across 
the software development lifecycle. A large part of the 
magic in DevOps comes from “doing rather than pro- 
crastinating,’ so agencies should start familiarizing them- 
selves with smaller and iterative approaches to software 
development combined with fully automated release pro- 
cesses—a state where everything moves with purpose. 


Even in situations where rigorous compliance testing is seen as a tough but 
necessary control point, DevOps and automation can pay dividends. By encap- 
sulating testing within development for example, DevOps practitioners can 
enact the necessary speed and quality improvements, while demonstrating 
reduced costs in auditing, risk management, and compliance. This is again very 
attractive to cash-constrained government departments. 
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e Untapped skills—When things are slow and hard to 
change, everything atrophies, including staff and their 
skills. If DevOps is to help slow moving government 
agencies, it's important to identify all issues that prevent 
change. Over years many staff will have become accus- 
tomed to working in a certain way, with incentives linked 
to these practices. It's important to maximize the motiva- 
tional impact of new DevOps programs, while also ensur- 
ing staff to stay the course. This may involve changes in 
organizational structures, new talent acquisition strate- 
gies (discussed earlier), and the adoption of shared goals, 
incentives, and metrics. 


Summary:The Seven Point Action Plan 


As we've discussed in this book, DevOps aims to break down the traditional 
barriers between development and other IT groups. By unifying people, tech- 
nology, and processes across a modern software factory, new applications and 
updates can be built, tested, and deployed more quickly. Together with con- 
tinuous feedback, businesses can enhance customer experiences, accelerate 
time to value, and better leverage IT as a competitive differentiator. 


But while interest in DevOps continues to grow, many implementations can 
fall short of expectations or stall completely. In part, and as we've described, 
this is because DevOps challenges conventional IT wisdom—placing less 
emphasis on process rigor and more on removing organizational barriers and 
continuous adjustment. 


As such, DevOps doesn't come pre-packaged with implementation blueprints; 
however, there are steps any organization (irrespective of DevOps maturity) 
can follow to optimize a DevOps initiative and we present these as a summary. 


Understand the Business Goal 


A program like DevOps is going take lots of work. It's great to get carried 
away with the “DevOps buzz,” but DevOps comes with a price tag, which 
includes new tools and processes to learn, people to cajole (and console), 
and teams to restructure.As with any business decision, all this cost and work 
must be carefully weighed against the expected benefits. 


Doing DevOps for the sake of DevOps is no way to start. Fall into this trap and 
customer value and business benefits take a backseat and become less impor- 
tant than protracted technology and process discussions. That's a DevOps 
disaster before you've even started. 
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It's important therefore to get business and IT on the same DevOps page. 


Focusing first on the needs of the business and then aligning DevOps, includ- 
ing metrics, tools, and processes. 


By focusing first and foremost on the needs of the business and clearly articu- 
lating concrete goals across IT, you'll ensure a consistent approach and avoid 
misinterpretations or DevOps misfires. 


As we like to say—don't fall in love with DevOps, fall in love with the business 
problem and let DevOps help you solve it! 


Cultivate Senior Level Sponsors 


If you're going to be changing things like organizational structures, work prac- 
tices, and incentives, you'll need some support. In large organizations, change 
takes time, so seek out a sponsor to facilitate decisions and make the tough 
calls. 


Executive level sponsors add substance to the DevOps sauce.They give teams 
a sense that there's real purpose behind the initiative and that it's not another 
process "wild goose chase". Remember too that senior managers are great 
at PR, which is very useful when you want to spread the good word about 
DevOps, build momentum, and gain more support. 


Note There's no such thing as a "free lunch". Executive sponsors expect real returns from 
DevOps, so try to make sure the goals being set can help them meet theirs. 


Select Your Pit-Crew 


People will make or break a DevOps initiative, so choosing the right staff is 


important.There are no hard and fast rules, but seek out employees who can: 


e Work well in teams—Look for people who have demon- 
strated a willingness to collaborate across organizational 
boundaries. 


Caution Be careful building a dedicated “DevOps team,” as this might become another silo. 
However, recognize that temporary teaming could be a good way of seeding DevOps goodness 
across the wider organization. 
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e Show resilience and flexibility—Great DevOps practitio- 
ners quickly learn from failure and embrace new ways of 
thinking to challenge the status quo. With developers for 
example, this can involve taking full ownership of produc- 
tion rollouts, not just the code they're working on. 


e Empathize with colleagues—Seek out staff across the orga- 
nization who consistently place themselves in the "shoes" 
of their colleagues. If a developer takes time to work with 
on-call staff to determine where supportability improve- 
ments are needed then they're a great candidate. 


Note Generalists are preferred over specialists, but strong consideration should be given to 
staff who have experience with newer practices, including continuous integration and testing, 
release automation, and toolset integrations. 


Revisit Chapter 3 and the section on DevOps culture for more pointers. 


Choose an Immediate Deliverable 


Select an application that's small enough to remain manageable with new 
DevOps approaches, but has enough business visibility to stimulate wider 
adoption. There's nothing to stop applying a DevOps approach to any type of 
application, but it's generally better suited to mobile and web facing applica- 
tions, which are architected to take advantage of aspects of DevOps tooling 
described in this book, including API management, continuous delivery, and 
"shift-left" monitoring. 


Legacy applications and systems-of-record can also benefit hugely from 
DevOps and shouldn't be ignored. Newer customer-facing mobile apps may 
need to integrate with back-end applications hosted on mainframes providing 
essential horsepower. Unfortunately, the practice of managing mainframes in 
silos can severely impact software development and adversely impact cus- 
tomer experience. 


Take, for example, a situation where a mobile development team needs access 
test data residing on a mainframe or insight into the performance implications 
of newly released code. These teams must have fast unconstrained access to 
this data to meet schedules together with end-to-end performance insights— 
mobile to mainframe. If teams work in silos using point tools, this will be dif- 
ficult to achieve. 


Many advanced tools, together with in agile-style methods, can help align 
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mainframe management with a DevOps initiative.This includes: 


Revisit Chapters 4-7, where these capabilities are described in greater detail. 
Always consider that tooling shouldn't be limited to managing one particular 


Extreme performance visibility—These tools provide deep 
visibility into complex interactions and transactions 
across mobile, web, and mainframe applications.They help 
teams quickly understand and troubleshoot even the old- 
est and most complex code bases. 


Constraint removal —By simulating mainframe infrastruc- 
ture dependencies and providing synthetic test data, 
developers and testers can maintain the pace of DevOps 
delivery without compromising quality or compliance 
goals. 


Unified infrastructure management—Enables mainframe and 
non-mainframe experts alike to better understand infra- 
structure interdependencies, thereby detecting and fixing 
problems faster and without the need for specialist tools. 


Extensible analytics—By extending analytics into areas 
such as workload automation, teams can perform real-time 
forecasting of critical back-end services. Using advanced 
statistical methods to calculate job duration, administra- 
tors can fine-tune workload schedules and processing to 
cater to changing conditions. 


Exposing rich data sources—Allows developers to rapidly 
create application back-ends for internal applications, 
mobile apps, standalone microservices, data as a service, 
and partner integrations. These systems can help decom- 
pose large mainframe applications into self-contained 
units with everything needed for app delivery, including 
data integration, business logic, and a robust API layer. 


platform or supporting a single group. 


Build a Comprehensive Metrics Program 


Traditionally, and as we described in Chapter 3, goals and objectives have been 
set at a departmental level, with methods for measuring IT capability skewed 
toward resolving problems and rewarding the "heroes" who fix them. With 
DevOps, objectives should be set at a "singular team" level and aligned to 
the desired business end-state—digital transformation via faster release of 


high-quality software. 
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With this in mind, work collaboratively to build shared objectives and metrics 
aligned to business objectives. Avoid vanity metrics that distort the truth or 
incentives that reward siloed behaviors. Next, set some improvement targets 
and assess the capability of existing tools in meeting and surpassing your goals. 


Tip Just because a target is constantly being achieved doesn't mean improvements can't be 
made. Don't get complacent. Just like code, DevOps metrics need constant care and attention! 


Integrate DevOps with Existing Practices 


As we discussed in Chapter 8, DevOps doesn't preclude leveraging other 
methodologies and best practices, including ITIL. 


To avoid methodology “turf wars,” it'll take teams working collaboratively to 
identify where existing processes need to refining to cater to DevOps-style 
delivery. Obvious places to start are with existing release and change man- 
agement, but be prepared for "give and take" on both sides. Never forget 
that DevOps itself can benefit from well-established processes, such as inci- 
dent and problem management to strengthen feedback loops and accumulate 
organizational knowledge. 


Tip Consider establishing a single workflow-based communication process between 
developers and other groups, including support staff. Integrate these with monitoring, ticketing, 
and help desk solutions to make problems and solutions visible across the entire organization. 


To be fully effective, DevOps practitioners also need to work with many 
other stakeholders, including enterprise architects and security. Once again, 
and using the guidance described in Chapter 8, be prepared to adjust existing 
methods and take other perspectives on board. 


Automate the Pipeline; Strengthen the Toolchain 


Reexamine the "eight elements of waste" introduced in Chapter 3. It's a useful 
framework to better identify all the existing process and technology con- 
straints across your own software factory. 


While pushing code all the way to production via fully automated releases 
might be the holy grail of DevOps, it could be better to start by reviewing 
those areas experiencing the greatest pain or where integrated toolsets can 
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help fill in the release pipeline "whitespace" (for examples, revisit the release 
automation integration use cases illustrated in Chapter 6 and the automated 
testing trifecta presented in Chapter 5). 


In other cases, reviewing waste and constraints may indicate that operational 
functions have the most to benefit from automation in the short term. This 
could include the adoption of modern configuration methods (infrastructure 
as code), or establishing performance monitoring and analytics to build criti- 
cal bidirectional feedback loops between development and IT operations (for 
an illustrative example, loop back to case study and discussion on “shift-left” 
monitoring presented in Chapter 7). 


Tip Asintraditional manufacturing, many wasteful IT practices and constraints inhibit the flow 
of value/throughput. Identify the greatest painpoint and potential for human error, be it managing 
test data, accessing infrastructure dependencies, or conducting manual and error-prone handoffs. 


In Salute of DevOps 


In a world where customers and citizens expect so much more from technol- 


ogy, the race is on to develop a finely tuned high-performance IT function. 


Hopefully by reading this book you're in a better position to see how DevOps 
supports this imperative and can develop the strategies needed to reignite 
your business. 


In Formula | racing, winning one race doesn't win a championship. It’s the 
same in software development. DevOps teams understand this implicitly. They 
are smashing silos, collaborating fully, and leveraging advanced automated solu- 
tions to eke out business performance improvements everywhere, as they 
build, test, release, and manage.... 


.... superior high-quality software solutions, continuously delivered from a 
modern DevOps-enabled software factory. 


....Start your engines! 
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